You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2021/08/18 18:04:25 UTC
[GitHub] [trafficcontrol] TaylorCFrey opened a new pull request #6118: Feature/refetch
TaylorCFrey opened a new pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118
<!--
Thank you for contributing! Please be sure to read our contribution guidelines: https://github.com/apache/trafficcontrol/blob/master/CONTRIBUTING.md
If this closes or relates to an existing issue, please reference it using one of the following:
Closes: #ISSUE
Related: #ISSUE
If this PR fixes a security vulnerability, DO NOT submit! Instead, contact
the Apache Traffic Control Security Team at security@trafficcontrol.apache.org and follow the
guidelines at https://apache.org/security regarding vulnerability disclosure.
-->
This PR is to merge the changes necessary to implement the Traffic Ops related changes for the Refetch Blueprint ([PR 5910](https://github.com/apache/trafficcontrol/pull/5910)).
The changes primarily fall on schema changes for the database and Traffic Ops API changes. Currently this is targeting Traffic Ops API 4.0.
TO Database:
- Remove unused tables and columns (_job\_status_ table, _job\_agent_ table, various columns in the _job_ table)
- Modify the data within the _job_ table to reflect the new schema and goal (e.g. remove the _keyword_ column and change the _parameters_ column to _ttl\_hr_, etc.)
- TODO/Outstanding: Update `app/db/seeds.sql` and `app/db/patches.sql` to account for db schema changes
TO API:
- Add v4.0 InvalidationJob endpoints and structs to accomplish the requirements in the _blueprint/refetch_ goals.
- Modify the previous endpoints to ensure compatibility with the DB schema changes
TO clients:
- Modify the v4 clients to take advantage of the TO v4.0 API and tc structs
- Ensure the tests pass
TP:
- Modify TP to use the _stable_ (3.1) version of the /jobs endpoint
T3C (formerly ORT):
- Slight modification to ensure tests pass with the DB schema
Documentation:
- TODO/Outstanding: Want to ensure we are actually going to merge this into v4.0 before modifying all the documentation to reflect this.
<!-- **^ Add meaningful description above** --><hr>
## Which Traffic Control components are affected by this PR?
<!-- Please delete all components from this list that are NOT affected by this PR.
Feel free to add the name of a tool or script that is affected but not on the list.
-->
- Documentation
- Traffic Control Cache Config (T3C, formerly ORT)
- Traffic Control Client <!-- Please specify which (Python, Go, or Java) -->
- Traffic Ops
- Traffic Portal
## What is the best way to verify this PR?
<!-- Please include here ALL the steps necessary to test your PR.
If your PR has tests (and most should), provide the steps needed to run the tests.
If not, please provide step-by-step instructions to test the PR manually and explain why your PR does not need tests. -->
The TO database migration scripts will need to be run to ensure the transition from a previous schema works to the new. This is accomplished by using the `db/admin` utility as outlined [here](https://traffic-control-cdn.readthedocs.io/en/latest/development/traffic_ops.html#app-db-admin).
Once the DB migration has taken place, the Traffic Ops client API integration tests can be run. You will need TO to be up and running with the _to\_test_ database set up as described [here](https://traffic-control-cdn.readthedocs.io/en/latest/development/traffic_ops.html#test-cases).
- To test all currently supported APIs (ensures backwards compatibility as well) you can do something akin to:
```bash
cd github.com/apache/trafficcontrol/traffic_ops/testing/api
# default location is github.com/apache/trafficcontrol/traffic_ops/testing/api/conf/traffic-ops-test.conf
go test --cfg <location for your test config> ./...
```
- To test only the invalidation job specific changes for API 4.0 you could run something akin to:
```bash
cd github.com/apache/trafficcontrol/traffic_ops/testing/api/v4
# default location is github.com/apache/trafficcontrol/traffic_ops/testing/api/conf/traffic-ops-test.conf
go test -run Jobs --cfg <location for your test config>
```
Once the Traffic Ops Integration tests confirm the new and previous implementations of the _/jobs_ endpoints still work then you can proceed to running and testing the TP integration tests still work. This will verify that TP is utilizing the previous stable version of the TO API (v3.1) and not the new _/jobs_ endpoints.
Ensure a local version of TO is up and running with the correct DB schema. This should already be the case based on the TO client integration tests from before.
Ensure Traffic Portal is building and running as outline (here)[https://traffic-control-cdn.readthedocs.io/en/latest/development/traffic_portal.html#traffic-portal].
> By default, the TP integration tests are set up to run against CIAB. If you're running this locally you will need to modify the config file located at github.com/apache/trafficcontrol/traffic_portal/test/integration/config.json. The `params` section will need to be modified:
```json
"params": {
"apiUrl": " https://localhost:8443/api/4.0",
"baseUrl": "https://localhost:9090",
"login": {
"username": "admin",
"password": "twelve12"
},
```
> Where
apiUrl = Running Traffic Ops API URL (This can and should be set to 4.0)
baseUrl = Running Traffic Portal URL
login.username = existing administrator account username
login.password = existing administrator account password
You can also, optionally, remove the `--headless` flag to see the integration tests as they run in Chrome Browser
```bash
cd github.com/apache/trafficcontrol/traffic_portal/test/integration
npm i
npm run start-webdriver
npm run test
```
## PR submission checklist
- [x] This PR has tests <!-- If not, please delete this text and explain why this PR does not need tests. -->
- [ ] This PR has documentation - This is delayed for the time being until accepted or rejected in the current RC (6.0) for ATC.
- [ ] This PR has a CHANGELOG.md entry - This is delayed for the time being until accepted or rejected in the current RC (6.0) for ATC.
- [x] This PR **DOES NOT FIX A SERIOUS SECURITY VULNERABILITY** (see [the Apache Software Foundation's security guidelines](https://apache.org/security) for details)
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r697667802
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.up.sql
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Migrate
+
+/**
+ * Alter job_agent table and data
+ */
+
+DROP TABLE IF EXISTS public.job_agent CASCADE;
+
+/**
+ * Alter job_status table and data
+ */
+
+DROP TABLE IF EXISTS public.job_status CASCADE;
+
+/**
+ * Alter job table and data
+ */
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS agent CASCADE,
+DROP COLUMN IF EXISTS object_type CASCADE,
+DROP COLUMN IF EXISTS object_name CASCADE,
+DROP COLUMN IF EXISTS keyword CASCADE,
+DROP COLUMN IF EXISTS asset_type CASCADE,
+DROP COLUMN IF EXISTS status CASCADE;
+
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS invalidation_type text NOT NULL DEFAULT 'REFRESH';
+
+/*
+ * If the asset_url contains the temporary fix for refetch
+ * (adding ##REFETCH## to the end of the url) then assign the
+ * correct invalidation_type.
+ */
+UPDATE public.job
+SET invalidation_type = 'REFETCH'
+WHERE id IN (
+ SELECT id
+ FROM public.job
+ WHERE asset_url LIKE '%##REFETCH##'
+);
Review comment:
Good catch. Simplifying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725312153
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -270,6 +475,12 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
return nil, nil, fmt.Errorf("parsing db response: %v", err), http.StatusInternalServerError, nil
}
+ // NamedQuery (nor prepare statements) cannot handle string functions in SELECT such as:
+ // CONCAT('TTL:', ttl_hr, 'h') AS parameters
Review comment:
Is the thought that the `:` isn't escaped and is somehow parsed by the `NamedQuery` func which is causing an issue? Does two `:` alleviate this? I'll have to look into this.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
+
+ accessibleTenants, err := tenant.GetUserTenantIDListTx(job.APIInfo().Tx.Tx, job.APIInfo().User.TenantID)
+ if err != nil {
+ return nil, nil, fmt.Errorf("getting accessible tenants for user - %v", err), http.StatusInternalServerError, nil
+ }
+ if len(where) > 0 {
+ where += " AND ds.tenant_id = ANY(:tenants) "
+ } else {
+ where = dbhelpers.BaseWhere + " ds.tenant_id = ANY(:tenants) "
+ }
+ queryValues["tenants"] = pq.Array(accessibleTenants)
+
+ if useIMS {
+ runSecond, maxTime = ims.TryIfModifiedSinceQuery(job.APIInfo().Tx, h, queryValues, selectMaxLastUpdatedQuery(where))
+ if !runSecond {
+ log.Debugln("IMS HIT")
+ return []interface{}{}, nil, nil, http.StatusNotModified, &maxTime
+ }
+ log.Debugln("IMS MISS")
+ } else {
+ log.Debugln("Non IMS request")
+ }
+
+ query := readQueryV40 + where + orderBy + pagination
+ log.Debugln("generated job query: " + query)
+ log.Debugf("executing with values: %++v\n", queryValues)
+
+ returnable := []interface{}{}
+ rows, err := job.APIInfo().Tx.NamedQuery(query, queryValues)
+ if err != nil {
+ return nil, nil, fmt.Errorf("querying: %v", err), http.StatusInternalServerError, nil
+ }
+ defer rows.Close()
+
+ for rows.Next() {
+ job := tc.InvalidationJobV40{}
+ err := rows.Scan(&job.ID,
+ &job.AssetURL,
+ &job.CreatedBy,
+ &job.DeliveryService,
+ &job.TTLHours,
+ &job.InvalidationType,
+ &job.StartTime)
+ if err != nil {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726609411
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
We haven't always been the best at sticking to major-version only while hiding most minor-version details. In general, just think about the amount of code that needs to change when we inevitably add another minor version. You want the blast-radius to be as small as possible when that happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r717005589
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -429,6 +787,198 @@ func Create(w http.ResponseWriter, r *http.Request) {
// Used by PUT requests to `/jobs`, replaces an existing content invalidation job
// with the provided request body.
+func UpdateV40(w http.ResponseWriter, r *http.Request) {
+ inf, userErr, sysErr, errCode := api.NewInfo(r, nil, nil)
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ var oFQDN string
+ var dsid uint
+ var uid uint
+ job := tc.InvalidationJobV40{}
+ row := inf.Tx.Tx.QueryRow(putInfoQueryV40, inf.Params["id"])
Review comment:
I'm thinking `id` needs to be a `requiredParam` and `intParam` in the call to `api.NewInfo`.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
I completed this after you started this PR, but please ensure that [my changes](https://github.com/apache/trafficcontrol/pull/6154/files#diff-3310237988f25e4cdf0632f9d8be54da09f3ac609ac7d8e8ba4dd74254d36bec) made to the prior API versions is replicated in API v4. We need the ability to filter the query by `cdn` and `maxRevalDurationDays`.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
I completed this after you started this PR, but please ensure that [my changes](https://github.com/apache/trafficcontrol/pull/6154/files#diff-3310237988f25e4cdf0632f9d8be54da09f3ac609ac7d8e8ba4dd74254d36bec) made to the prior API versions are replicated here in API v4. We need the ability to filter the query by `cdn` and `maxRevalDurationDays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727575214
##########
File path: traffic_ops/testing/api/v4/tc-fixtures.json
##########
@@ -5462,20 +5462,23 @@
{
"deliveryService": "ds1",
"regex": "/.*",
- "startTime": 4117118271000,
- "ttl": "121m"
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 72,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds1",
"regex": "/foo",
- "startTime": 4117118271000,
- "ttl": 2160
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 36,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds2",
"regex": "\\/some-path?.+\\.jpg",
- "startTime": "2100-06-19T13:57:51-06:00",
- "ttl": 2.1
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 144,
+ "invalidationType" : "REFRESH"
Review comment:
I agree we need to add some `REFETCH` jobs. The order of operations with requiring a parameter value makes it a little difficult to add to the tc-fixtures file though. I will work on adding these in the go test file explicitly.
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725321923
##########
File path: cache-config/t3cutil/toreq/clientfuncs.go
##########
@@ -513,24 +513,24 @@ func (cl *TOClient) GetDeliveryServiceRegexes(reqHdr http.Header) ([]tc.Delivery
return regexes, reqInf, nil
}
-func (cl *TOClient) GetJobs(reqHdr http.Header) ([]tc.InvalidationJob, toclientlib.ReqInf, error) {
+func (cl *TOClient) GetJobs(reqHdr http.Header) ([]tc.InvalidationJobV40, toclientlib.ReqInf, error) {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-957865350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725323083
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r717017592
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
I completed this after you started this PR, but please ensure that [my changes](https://github.com/apache/trafficcontrol/pull/6154/files#diff-3310237988f25e4cdf0632f9d8be54da09f3ac609ac7d8e8ba4dd74254d36bec) made to the prior API versions are replicated here in API v4. We need the ability to filter the query by `cdn` and `maxRevalDurationDays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725241687
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
Done. However, I do see V40 for `DeliveryServiceRequestV40` and `ServerV40`, but then I only see the major versions for other object like `RoleV4` and `UserV4`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725327849
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725341990
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Excellent point. Though, to add to this, aren't the status values just as relative to whomever is managing their CDN system? Or do we require that a job status of `PENDING` exist in all installations of TO/ATC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727431974
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
Review comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724574893
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
Good points! Should I move this, then, into _traffic\_ops\_golang_? I was mirroring the implementation of the previous `func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error` function, but extracting the db calls into separate funcs. If I move the db calls into, say, `dbhelpers` then I think I would be stuck as to whether the logic should reside in the the _/lib/go-tc_ or in _traffic\_ops\_golang_
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
This is great! Ok, so...
1. Move it to something specific like `regex-revalidation.config`, rather than `global`
2. Change the query to something that actually returns a boolean value like
```sql
SELECT 'true' = lower(trim(p.value)) FROM parameter p WHERE [...]
```
to avoid any coercion issues. This still won't entirely prevent bad input as the value (say the user enters `foo` in the hopes that it is a truthy/falsy style comparison)
3. The previous DB invocations in `lib/tc-go/invdliationjobs.go` would log DB errors, which would at least note the error and prevent it from being swallowed, but still not leak DB errors to the user. (See previous comment about moving all DB calls into _traffic\_ops\_golang_)
4. Lastly, If having multiple rows returned per possible configuration value in the table were a problem, then it would be a problem for all configuration values I assume. If that's the case, should we add a unique multi-column index to the `name` and `config_file` column to prevent any possible duplicates across the board? I'm not sure if that would cause issues with other, preexisting values or not.
Did I understand that correctly? Should I proceed with all 4 recommendations or any ones in particular?
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
+}
Review comment:
I'm not sure what is meant here. My attempt was to mirror the previous implementations to keep it as in-line as before. However if we are moving towards a different approach could you and I sit down and discuss this?
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -282,6 +493,152 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
// Used by POST requests to `/jobs`, creates a new content invalidation job
// from the provided request body.
+func CreateV40(w http.ResponseWriter, r *http.Request) {
+ inf, userErr, sysErr, errCode := api.NewInfo(r, nil, nil)
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ job := tc.InvalidationJobCreateV40{}
+ if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New("Unable to parse Invalidation Job"), fmt.Errorf("parsing jobs/ POST: %v", err))
+ return
+ }
+
+ // Check if request object is valid
+ w.Header().Set(rfc.ContentType, rfc.ApplicationJSON)
+ if err := job.Validate(inf.Tx.Tx); err != nil {
+ response := tc.Alerts{
+ Alerts: []tc.Alert{
+ {
+ Text: err.Error(),
+ Level: tc.ErrorLevel.String(),
+ },
+ },
+ }
+
+ resp, err := json.Marshal(response)
+ if err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusInternalServerError, nil, fmt.Errorf("Encoding bad request response: %v", err))
+ return
+ }
+
+ w.WriteHeader(http.StatusBadRequest)
+ api.WriteAndLogErr(w, r, append(resp, '\n'))
Review comment:
Will change!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724587524
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
+}
Review comment:
I'm not sure what is meant here. My attempt was to mirror the previous implementations to keep it as in-line as before. However if we are moving towards a different approach could you and I sit down and discuss this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727577725
##########
File path: lib/go-atscfg/atscfg.go
##########
@@ -655,12 +654,6 @@ func makeErr(warnings []string, err string) error {
return errors.New(`(warnings: ` + strings.Join(warnings, `, `) + `) ` + err)
}
-// makeErrf is a convenience for formatting errors for makeErr.
-// todo also unused, maybe remove?
-func makeErrf(warnings []string, format string, v ...interface{}) error {
Review comment:
I would prefer we didn't remove this. It isn't used at the moment, but it's generally useful as a library function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r692496032
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.up.sql
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Migrate
+
+/**
+ * Alter job_agent table and data
+ */
+
+DROP TABLE IF EXISTS public.job_agent CASCADE;
+
+/**
+ * Alter job_status table and data
+ */
+
+DROP TABLE IF EXISTS public.job_status CASCADE;
+
+/**
+ * Alter job table and data
+ */
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS agent CASCADE,
+DROP COLUMN IF EXISTS object_type CASCADE,
+DROP COLUMN IF EXISTS object_name CASCADE,
+DROP COLUMN IF EXISTS keyword CASCADE,
+DROP COLUMN IF EXISTS asset_type CASCADE,
+DROP COLUMN IF EXISTS status CASCADE;
+
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS invalidation_type text NOT NULL DEFAULT 'REFRESH';
+
+/*
+ * If the asset_url contains the temporary fix for refetch
+ * (adding ##REFETCH## to the end of the url) then assign the
+ * correct invalidation_type.
+ */
+UPDATE public.job
+SET invalidation_type = 'REFETCH'
+WHERE id IN (
+ SELECT id
+ FROM public.job
+ WHERE asset_url LIKE '%##REFETCH##'
+);
Review comment:
I don't think you need this nested query. You can just take the `WHERE` clause from the nested query and use that as the `WHERE` clause for your `UPDATE` query.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -282,6 +493,152 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
// Used by POST requests to `/jobs`, creates a new content invalidation job
// from the provided request body.
+func CreateV40(w http.ResponseWriter, r *http.Request) {
+ inf, userErr, sysErr, errCode := api.NewInfo(r, nil, nil)
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ job := tc.InvalidationJobCreateV40{}
+ if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New("Unable to parse Invalidation Job"), fmt.Errorf("parsing jobs/ POST: %v", err))
+ return
+ }
+
+ // Check if request object is valid
+ w.Header().Set(rfc.ContentType, rfc.ApplicationJSON)
+ if err := job.Validate(inf.Tx.Tx); err != nil {
+ response := tc.Alerts{
+ Alerts: []tc.Alert{
+ {
+ Text: err.Error(),
+ Level: tc.ErrorLevel.String(),
+ },
+ },
+ }
+
+ resp, err := json.Marshal(response)
+ if err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusInternalServerError, nil, fmt.Errorf("Encoding bad request response: %v", err))
+ return
+ }
+
+ w.WriteHeader(http.StatusBadRequest)
+ api.WriteAndLogErr(w, r, append(resp, '\n'))
Review comment:
This should probably just be a one-liner that does the appending for you:
```go
api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, nil, nil)
```
Honestly I don't think `api.WriteAndLogErr` should even be exported
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
well it wasn't an unknown error, we just dropped it. This should return that error. I think the reason you're not doing that is because you don't want to expose database errors to the client, and you're returning an error that tells that the DS doesn't exist which will be shown to the client. This is why `dbhelpers` functions typically return _three_ values for things like this - the requested value, whether or not it could be found, and whatever database error might have occurred, which is assumed to be unsafe for the client to view.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
+}
Review comment:
Are we using the CRUDer? I thought we wanted to move away from that as a project in favor of middlewares and expanding `APIInfo` structure capabilities?
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
I assume that this won't always fail - which is what I would have guessed, but I'm sure you'd have noticed that -, but it for sure can be made to fail. Parameter values are strings, not booleans, so even if this coerces values `'true'` and `'false'` to booleans, it will for sure fail if the value of the Parameter is something like `'foo'`, which means that you can make a Parameter that's perfectly valid that will break an API endpoint so that it only ever returns 500 responses.
We also definitely don't want to be swallowing that error.
Further, the problem with this is that there is no validation on Parameters, and that WHERE clause can even match multiple Parameters. Plus, by convention, we put global Parameters in the GLOBAL Profile - but that Profile could also be deleted or renamed at any time.
So if I have two Parameters:
```json
[{
"name": "refetch_enabled",
"configFile": "global",
"value": "true",
"secure": false,
"id": 1
},
{
"name": "refetch_enabled",
"configFile": "global",
"value": "false",
"secure": false,
"id": 2
}]
```
Then the one chosen by this method is undefined, even if one of them shows up when you look at the GLOBAL profile, the other still might get chosen.
For all these reasons, I tend to prefer using configuration file parameters (i.e. in "cdn.conf"), whose values can be exposed through the `system/info` endpoint if other components need to know about them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725322043
##########
File path: lib/go-atscfg/atscfg.go
##########
@@ -696,3 +696,35 @@ func JobToInvalidationJob(jb tc.Job) (tc.InvalidationJob, error) {
StartTime: &startTime,
}, nil
}
+
+func JobsToInvalidationJobsV40(oldJobs []tc.Job) ([]tc.InvalidationJobV40, error) {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725185766
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -197,25 +179,13 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
if tcJob.StartTime.Add(ttl).Before(time.Now()) {
continue
}
- if tcJob.Keyword == nil || *tcJob.Keyword != JobKeywordPurge {
- continue
- }
-
if tcJob.AssetURL == nil {
continue
}
// process the __REFETCH__ keyword
- assetURL := *tcJob.AssetURL
- var jobType string
-
- if strings.HasSuffix(assetURL, RefetchSuffix) {
- assetURL = strings.TrimSuffix(assetURL, RefetchSuffix)
- jobType = "MISS"
- } else if strings.HasSuffix(assetURL, RefreshSuffix) { // also default
- assetURL = strings.TrimSuffix(assetURL, RefreshSuffix)
- jobType = "STALE"
- }
+ // There is no InvalidationType string before TO v4.0, so pass empty string
Review comment:
Nice catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r730145415
##########
File path: traffic_ops/testing/api/v4/jobs_test.go
##########
@@ -16,38 +16,40 @@ package v4
*/
import (
+ "fmt"
"net/http"
"strconv"
"strings"
"testing"
"time"
"github.com/apache/trafficcontrol/lib/go-tc"
- "github.com/apache/trafficcontrol/lib/go-util"
client "github.com/apache/trafficcontrol/traffic_ops/v4-client"
)
func TestJobs(t *testing.T) {
WithObjs(t, []TCObj{CDNs, Types, Tenants, Parameters, Profiles, Statuses, Divisions, Regions, PhysLocations, CacheGroups, Servers, Topologies, ServiceCategories, DeliveryServices}, func() {
- CreateTestJobs(t)
- CreateTestInvalidationJobs(t)
- CreateTestInvalidJob(t)
- GetTestJobsQueryParams(t)
- GetTestJobs(t)
- GetTestInvalidationJobs(t)
- JobCollisionWarningTest(t)
- GetTestJobsByValidData(t)
- GetTestJobsByInvalidData(t)
- CreateTestJobsInvalidDS(t)
- CreateTestJobsAlreadyExistTTL(t)
- CreateTestJobsWithPastDate(t)
- CreateTestJobsWithFutureDate(t)
- CreateJobsMissingDate(t)
- CreateJobsMissingRegex(t)
- CreateJobsMissingTtl(t)
- UpdateTestJobsInvalidDS(t)
- DeleteTestJobs(t)
- DeleteTestJobsByInvalidId(t)
+ // CreateTestJobs(t)
+ // CreateTestInvalidationJobs(t)
+ // CreateTestInvalidJob(t)
+ // GetTestJobsQueryParams(t)
+ // GetTestJobs(t)
+ // GetTestInvalidationJobs(t)
+ // JobCollisionWarningTest(t)
+ // GetTestJobsByValidData(t)
+ // GetTestJobsByInvalidData(t)
+ // CreateTestJobsInvalidDS(t)
+ // CreateTestJobsAlreadyExistTTL(t)
+ // CreateTestJobsWithPastDate(t)
+ // CreateTestJobsWithFutureDate(t)
+ // CreateJobsMissingDate(t)
+ // CreateJobsMissingRegex(t)
+ // CreateJobsMissingTtl(t)
+ // UpdateTestJobsInvalidDS(t)
+ // DeleteTestJobs(t)
+ // DeleteTestJobsByInvalidId(t)
Review comment:
Looks like you accidentally left these commented out
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -331,83 +715,269 @@ func Create(w http.ResponseWriter, r *http.Request) {
return
}
- w.WriteHeader(http.StatusBadRequest)
- api.WriteAndLogErr(w, r, append(resp, '\n'))
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New(string(append(resp, '\n'))), fmt.Errorf("error validating job: %v", err))
Review comment:
I don't think this is right. This code is putting the response in the `userErr` field and the `sysErr` is the error marshalling the response. I think the original code here was correct (`w.WriteHeader` followed by `api.WriteAndLogErr`.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -331,83 +715,269 @@ func Create(w http.ResponseWriter, r *http.Request) {
return
}
- w.WriteHeader(http.StatusBadRequest)
- api.WriteAndLogErr(w, r, append(resp, '\n'))
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New(string(append(resp, '\n'))), fmt.Errorf("error validating job: %v", err))
Review comment:
I don't think this is right. This code is putting the response in the `userErr` field and the `sysErr` is the error marshalling the response. I think the original code here was correct (`w.WriteHeader` followed by `api.WriteAndLogErr`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725327181
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
Review comment:
Done, but will need to refactor this into the `traffic_ops_golang` package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725313125
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -130,11 +156,35 @@ WHERE server.status NOT IN (
const updateQuery = `
UPDATE job
SET asset_url=$1,
- keyword=$2,
- parameters=$3,
- start_time=$4
+ ttl_hr=$2,
+ start_time=$3
+WHERE job.id=$4
+RETURNING asset_url,
+ (
+ SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS delivery_service,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ start_time
+`
+
+// Almost the same as updateQuery, but returns appropriate values for API 4.0+
+const updateQueryV40 = `
Review comment:
Done. However, there is an instance of precedence (`traffic_ops_golang/user`) where these functions are v40. Perhaps these should also be v4 as a separate issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727570351
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-957865350
I think the DB migrations may need to be updated to a more recent timestamp: ```traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql```
Because there have been other migrations that have come in with more recent timestamps, this one won't be run in dev environments that have already run those more recent migrations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp merged pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp merged pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r734977828
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -331,83 +715,269 @@ func Create(w http.ResponseWriter, r *http.Request) {
return
}
- w.WriteHeader(http.StatusBadRequest)
- api.WriteAndLogErr(w, r, append(resp, '\n'))
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New(string(append(resp, '\n'))), fmt.Errorf("error validating job: %v", err))
Review comment:
So, in general, when handling an error, most code will just call `api.HandleErr` with the proper status code, user error, and system error. That function handles creating and marshalling the JSON response.
Calling `api.HandleErr` _here_ was incorrect because the code just above it manually created and marshalled the JSON response already. Therefore, it makes sense to leave the `api.WriteAndLogErr` as-is here, since the response was already marshalled and ready to write. However, this code probably shouldn't be manually creating the response+alerts itself. After calling `job.Validate`, it should probably just be calling `api.HandleErr` if there was a validation error; it doesn't need to manually handle the marshalling error because `api.HandleErr` does that already.
That said, I don't think that needs to be corrected in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725344334
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
I will! I saw those come through and will double check those changes are in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725343000
##########
File path: traffic_ops/testing/api/v4/jobs_test.go
##########
@@ -58,19 +57,17 @@ func CreateTestJobs(t *testing.T) {
}
for i, job := range testData.InvalidationJobs {
- job.StartTime = &tc.Time{
- Time: time.Now().Add(time.Minute).UTC(),
- Valid: true,
- }
+ job.StartTime = time.Now().Add(time.Minute).UTC()
testData.InvalidationJobs[i] = job
}
for _, job := range testData.InvalidationJobs {
- request := tc.InvalidationJobInput{
- DeliveryService: job.DeliveryService,
- Regex: job.Regex,
- StartTime: job.StartTime,
- TTL: job.TTL,
+ request := tc.InvalidationJobCreateV40{
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725321740
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -486,7 +491,7 @@ func MakeFakeTOData() *t3cutil.ConfigData {
},
DeliveryServiceServers: dss,
Server: sv0,
- Jobs: []tc.InvalidationJob{
+ Jobs: []tc.InvalidationJobV40{
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725312326
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725244346
##########
File path: lib/go-util/ptr.go
##########
@@ -1,5 +1,7 @@
package util
+import "time"
Review comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725313668
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-957865350
I think the DB migrations may need to be updated to a more recent timestamp: ```traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql```
Because there have been other migrations that have come in with more recent timestamps, this one won't be run in dev environments that have already run those more recent migrations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r717005589
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -429,6 +787,198 @@ func Create(w http.ResponseWriter, r *http.Request) {
// Used by PUT requests to `/jobs`, replaces an existing content invalidation job
// with the provided request body.
+func UpdateV40(w http.ResponseWriter, r *http.Request) {
+ inf, userErr, sysErr, errCode := api.NewInfo(r, nil, nil)
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ var oFQDN string
+ var dsid uint
+ var uid uint
+ job := tc.InvalidationJobV40{}
+ row := inf.Tx.Tx.QueryRow(putInfoQueryV40, inf.Params["id"])
Review comment:
I'm thinking `id` needs to be a `requiredParam` and `intParam` in the call to `api.NewInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726611554
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -270,6 +475,12 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
return nil, nil, fmt.Errorf("parsing db response: %v", err), http.StatusInternalServerError, nil
}
+ // NamedQuery (nor prepare statements) cannot handle string functions in SELECT such as:
+ // CONCAT('TTL:', ttl_hr, 'h') AS parameters
Review comment:
Yes, that is my thought. I've run into this issue before, and I believe that was one of the solutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726551251
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
My bad. Given this is part of the "revert" migration, and we know there will be a `PENDING` value since it's being inserted, there exists a chance that the _indentity_ isn't reset and thus may not match the value of `1`. I will address this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727569803
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r717017592
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
I completed this after you started this PR, but please ensure that [my changes](https://github.com/apache/trafficcontrol/pull/6154/files#diff-3310237988f25e4cdf0632f9d8be54da09f3ac609ac7d8e8ba4dd74254d36bec) made to the prior API versions is replicated in API v4. We need the ability to filter the query by `cdn` and `maxRevalDurationDays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727570483
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
Most excellent. Ty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725242745
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
I believe that is a completely valid location and can certainly be move there. However, how do we then keep from getting a circular dependency? If defining the structs is in the library, as is the validation, how do we keep the validating logic in the library but pull the db calls into the app?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725322861
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -246,6 +246,12 @@ func (job *InvalidationJobInput) Validate(tx *sql.Tx) error {
return nil
}
+type compareJob struct {
+ AssetURL *string
Review comment:
Dig it, Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724587136
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
This is great! Ok, so...
1. Move it to something specific like `regex-revalidation.config`, rather than `global`
2. Change the query to something that actually returns a boolean value like
```sql
SELECT 'true' = lower(trim(p.value)) FROM parameter p WHERE [...]
```
to avoid any coercion issues. This still won't entirely prevent bad input as the value (say the user enters `foo` in the hopes that it is a truthy/falsy style comparison)
3. The previous DB invocations in `lib/tc-go/invdliationjobs.go` would log DB errors, which would at least note the error and prevent it from being swallowed, but still not leak DB errors to the user. (See previous comment about moving all DB calls into _traffic\_ops\_golang_)
4. Lastly, If having multiple rows returned per possible configuration value in the table were a problem, then it would be a problem for all configuration values I assume. If that's the case, should we add a unique multi-column index to the `name` and `config_file` column to prevent any possible duplicates across the board? I'm not sure if that would cause issues with other, preexisting values or not.
Did I understand that correctly? Should I proceed with all 4 recommendations or any ones in particular?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727431924
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -183,9 +277,8 @@ RETURNING job.asset_url,
FROM deliveryservice
WHERE deliveryservice.id=job.job_deliveryservice
) AS deliveryservice,
- job.id,
- job.keyword,
- job.parameters,
+ ttl_hr,
+ job.invalidation_type,
Review comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725312473
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -168,11 +238,35 @@ INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
WHERE job.id=$1
`
+// Deprecated, only to be used with versions below 4.0
const deleteQuery = `
DELETE
FROM job
WHERE job.id=$1
RETURNING job.asset_url,
+ ( SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS deliveryservice,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time
+`
+
+// Almost the same as deleteQuery, but returns appropriate values for API 4.0+
+const deleteQueryV40 = `
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725243775
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
Review comment:
Explicit > implicit. Will change. There are other instances of logging information in this particular file. We could log these errors as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725322448
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -57,7 +57,7 @@ func MakeRegexRevalidateDotConfig(
server *Server,
deliveryServices []DeliveryService,
globalParams []tc.Parameter,
- jobs []tc.InvalidationJob,
+ jobs []tc.InvalidationJobV40,
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725362336
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
Ah! I think I am understand what you're saying now. I may gotten this incorrect in my blueprint. The request for a _feature flag_ of sorts is here: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648706299
Based on that, should this be a `cdn.conf` value or a `parameter`? I went with `parameter` because I assumed we wanted it to be toggleable at runtime and queryable. Also I wasn't aware of the `system/info` endpoint.
...
It appears that the `system/info` it is limited to querying the `parameters` table for the _global_ file. Looking into the `parameters` endpoint(s) there appears to be loose inclusions of other tables (`profiles`) as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724578710
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
The logic should also go into `traffic_ops_golang` IMO. Basically, `lib/go-tc` was only meant to be for the various things that needed to be shared among components -- primarily TO and the TO client (for the data structs). Over time, things like validation logic have snuck into `lib/go-tc`, but really validation logic belongs in `traffic_ops_golang` IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-902944300
> Biggest thing that stood out to me was the CRUDer rearing its ugly head again
This PR doesn't appear to be using the CRUDer any more than the _original_ `/jobs` implementation. It still uses it for `Read` it appears, but create, update, and delete are all as CRUDer-free as the prior version. I'm guessing the prior version functions were mostly just copied then modified to work with the new changes we want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726605444
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
I don't really think it's a big problem to have the feature flag as a parameter. If that's how it's currently implemented, I don't think it's worth it to reimplement it as an option in cdn.conf. While not the most ideal solution, it does have precedents, and it really is temporary in that eventually all ATS/t3c instances will support it and will no longer require a feature flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r734842375
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -331,83 +715,269 @@ func Create(w http.ResponseWriter, r *http.Request) {
return
}
- w.WriteHeader(http.StatusBadRequest)
- api.WriteAndLogErr(w, r, append(resp, '\n'))
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New(string(append(resp, '\n'))), fmt.Errorf("error validating job: %v", err))
Review comment:
I may have misused the `HandleErr` func here. For the time being I've reverted the change.
In the future, is it the appropriate call if I correct it? It appears, both ways set the header response, then write the json resp back to the client. Perhaps I should have made the user error more friendly since it gets returned as an alert rather than just the json resp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r734837677
##########
File path: traffic_ops/testing/api/v4/jobs_test.go
##########
@@ -16,38 +16,40 @@ package v4
*/
import (
+ "fmt"
"net/http"
"strconv"
"strings"
"testing"
"time"
"github.com/apache/trafficcontrol/lib/go-tc"
- "github.com/apache/trafficcontrol/lib/go-util"
client "github.com/apache/trafficcontrol/traffic_ops/v4-client"
)
func TestJobs(t *testing.T) {
WithObjs(t, []TCObj{CDNs, Types, Tenants, Parameters, Profiles, Statuses, Divisions, Regions, PhysLocations, CacheGroups, Servers, Topologies, ServiceCategories, DeliveryServices}, func() {
- CreateTestJobs(t)
- CreateTestInvalidationJobs(t)
- CreateTestInvalidJob(t)
- GetTestJobsQueryParams(t)
- GetTestJobs(t)
- GetTestInvalidationJobs(t)
- JobCollisionWarningTest(t)
- GetTestJobsByValidData(t)
- GetTestJobsByInvalidData(t)
- CreateTestJobsInvalidDS(t)
- CreateTestJobsAlreadyExistTTL(t)
- CreateTestJobsWithPastDate(t)
- CreateTestJobsWithFutureDate(t)
- CreateJobsMissingDate(t)
- CreateJobsMissingRegex(t)
- CreateJobsMissingTtl(t)
- UpdateTestJobsInvalidDS(t)
- DeleteTestJobs(t)
- DeleteTestJobsByInvalidId(t)
+ // CreateTestJobs(t)
+ // CreateTestInvalidationJobs(t)
+ // CreateTestInvalidJob(t)
+ // GetTestJobsQueryParams(t)
+ // GetTestJobs(t)
+ // GetTestInvalidationJobs(t)
+ // JobCollisionWarningTest(t)
+ // GetTestJobsByValidData(t)
+ // GetTestJobsByInvalidData(t)
+ // CreateTestJobsInvalidDS(t)
+ // CreateTestJobsAlreadyExistTTL(t)
+ // CreateTestJobsWithPastDate(t)
+ // CreateTestJobsWithFutureDate(t)
+ // CreateJobsMissingDate(t)
+ // CreateJobsMissingRegex(t)
+ // CreateJobsMissingTtl(t)
+ // UpdateTestJobsInvalidDS(t)
+ // DeleteTestJobs(t)
+ // DeleteTestJobsByInvalidId(t)
Review comment:
Gah! /ashamed. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r734977828
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -331,83 +715,269 @@ func Create(w http.ResponseWriter, r *http.Request) {
return
}
- w.WriteHeader(http.StatusBadRequest)
- api.WriteAndLogErr(w, r, append(resp, '\n'))
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New(string(append(resp, '\n'))), fmt.Errorf("error validating job: %v", err))
Review comment:
So, in general, when handling an error, most code will just call `api.HandleErr` with the proper status code, user error, and system error. That function handles creating and marshalling the JSON response.
Calling `api.HandleErr` _here_ was incorrect because the code just above it manually created and marshalled the JSON response already. Therefore, it makes sense to leave the `api.WriteAndLogErr` as-is here, since the response was already marshalled and ready to write. However, this code probably shouldn't be manually creating the response+alerts itself. After calling `job.Validate`, it should probably just be calling `api.HandleErr` if there was a validation error; it doesn't need to manually handle the marshalling error because `api.HandleErr` does that already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726612665
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -130,11 +156,35 @@ WHERE server.status NOT IN (
const updateQuery = `
UPDATE job
SET asset_url=$1,
- keyword=$2,
- parameters=$3,
- start_time=$4
+ ttl_hr=$2,
+ start_time=$3
+WHERE job.id=$4
+RETURNING asset_url,
+ (
+ SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS delivery_service,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ start_time
+`
+
+// Almost the same as updateQuery, but returns appropriate values for API 4.0+
+const updateQueryV40 = `
Review comment:
It is likely that those should be v4 as well, but I wouldn't worry about that right now. The idea is that we don't want an entirely new query when we add a new V41, for instance, since the query would be exactly the same except for some new fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725343284
##########
File path: traffic_ops/testing/api/v4/traffic_control_test.go
##########
@@ -56,5 +56,5 @@ type TrafficControl struct {
SteeringTargets []tc.SteeringTargetNullable `json:"steeringTargets"`
Serverchecks []tc.ServercheckRequestNullable `json:"serverchecks"`
Users []tc.UserV40 `json:"users"`
- InvalidationJobs []tc.InvalidationJobInput `json:"invalidationJobs"`
+ InvalidationJobs []tc.InvalidationJobCreateV40 `json:"invalidationJobs"`
Review comment:
Done (Nice catch on all of these)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725313263
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -98,11 +86,49 @@ RETURNING
(SELECT tm_user.username
FROM tm_user
WHERE tm_user.id=job_user) AS createdBy,
- keyword,
- parameters,
+ 'PURGE' AS keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
start_time
`
+// Almost the same as insertQuery, but returns appropriate values for API 4.0+
+const insertQueryV40 = `
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725328070
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725186272
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -232,3 +202,22 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
return newJobs, warnings
}
+
+const MISS = "MISS"
+const STALE = "STALE"
+
+// ProcessRefetch determines the type of Invalidation, returns the corresponding jobtype
+// and "cleans" the regex URL for the asset to be invalidated. REFETCH trumps REFRESH,
+// whether in the AssetURL or as InvalidationType
+func ProcessRefetch(invalidationType, assetURL string) (string, string) {
Review comment:
Indeed it should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726520505
##########
File path: cache-config/t3cutil/getdatacfg.go
##########
@@ -69,7 +69,7 @@ type ConfigData struct {
Server *atscfg.Server `json:"server,omitempty"`
// Jobs must be all Jobs on the server's CDN. May include jobs on other CDNs.
- Jobs []tc.InvalidationJob `json:"jobs,omitempty"`
+ Jobs []tc.InvalidationJobV40 `json:"jobs,omitempty"`
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727570260
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
Review comment:
Moved into `traffic_ops_golang`. Now log the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r715853606
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Why hardcode `1` here instead of `(SELECT id FROM status WHERE name = 'PENDING')`?
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -146,20 +196,40 @@ RETURNING job.asset_url,
WHERE deliveryservice.id=job.job_deliveryservice
) AS delivery_service,
job.id,
- job.keyword,
- job.parameters,
- job.start_time
+ ttl_hr,
+ start_time,
+ invalidation_type
`
+// Deprecated, only to be used with versions below 4.0
const putInfoQuery = `
SELECT job.id AS id,
tm_user.username AS createdBy,
job.job_user AS createdByID,
job.job_deliveryservice AS dsid,
deliveryservice.xml_id AS dsxmlid,
job.asset_url AS assetURL,
- job.parameters,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time AS start_time,
+ origin.protocol || '://' || origin.fqdn || rtrim(concat(':', origin.port), ':') AS OFQDN
+FROM job
+INNER JOIN origin ON origin.deliveryservice=job.job_deliveryservice AND origin.is_primary
+INNER JOIN tm_user ON tm_user.id=job.job_user
+INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
+WHERE job.id=$1
+`
+
+// Almost the same as putInfoQuery, but returns appropriate values for API 4.0+
+const putInfoQueryV40 = `
+SELECT job.id AS id,
+ tm_user.username AS createdBy,
+ job.job_user AS createdByID,
+ job.job_deliveryservice AS dsid,
+ deliveryservice.xml_id AS dsxmlid,
+ job.asset_url AS assetURL,
+ job.ttl_hr AS ttlhrs,
job.start_time AS start_time,
+ job.invalidation_type as invalidationType,
Review comment:
formatting is off
##########
File path: traffic_ops/testing/api/v4/traffic_control_test.go
##########
@@ -56,5 +56,5 @@ type TrafficControl struct {
SteeringTargets []tc.SteeringTargetNullable `json:"steeringTargets"`
Serverchecks []tc.ServercheckRequestNullable `json:"serverchecks"`
Users []tc.UserV40 `json:"users"`
- InvalidationJobs []tc.InvalidationJobInput `json:"invalidationJobs"`
+ InvalidationJobs []tc.InvalidationJobCreateV40 `json:"invalidationJobs"`
Review comment:
This should use an `InvalidationJobCreateV4` alias as well.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
Review comment:
V40 -> V4 alias
##########
File path: traffic_ops/testing/api/v4/jobs_test.go
##########
@@ -58,19 +57,17 @@ func CreateTestJobs(t *testing.T) {
}
for i, job := range testData.InvalidationJobs {
- job.StartTime = &tc.Time{
- Time: time.Now().Add(time.Minute).UTC(),
- Valid: true,
- }
+ job.StartTime = time.Now().Add(time.Minute).UTC()
testData.InvalidationJobs[i] = job
}
for _, job := range testData.InvalidationJobs {
- request := tc.InvalidationJobInput{
- DeliveryService: job.DeliveryService,
- Regex: job.Regex,
- StartTime: job.StartTime,
- TTL: job.TTL,
+ request := tc.InvalidationJobCreateV40{
Review comment:
To echo a prior comment, the v4 TO API tests should be using the v4 alias job type `InvalidationJobCreateV4`, `InvalidationJobV4`, etc., as opposed to the v40 type.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -98,11 +86,49 @@ RETURNING
(SELECT tm_user.username
FROM tm_user
WHERE tm_user.id=job_user) AS createdBy,
- keyword,
- parameters,
+ 'PURGE' AS keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
start_time
`
+// Almost the same as insertQuery, but returns appropriate values for API 4.0+
+const insertQueryV40 = `
Review comment:
We should probably call this `V4` instead of `V40`, because we'll likely just update this query with new fields if they're ever added.
##########
File path: traffic_ops/testing/api/v4/tc-fixtures.json
##########
@@ -5462,20 +5462,23 @@
{
"deliveryService": "ds1",
"regex": "/.*",
- "startTime": 4117118271000,
- "ttl": "121m"
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 72,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds1",
"regex": "/foo",
- "startTime": 4117118271000,
- "ttl": 2160
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 36,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds2",
"regex": "\\/some-path?.+\\.jpg",
- "startTime": "2100-06-19T13:57:51-06:00",
- "ttl": 2.1
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 144,
+ "invalidationType" : "REFRESH"
Review comment:
We should add some `REFETCH`-type jobs to the tests as well.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -146,20 +196,40 @@ RETURNING job.asset_url,
WHERE deliveryservice.id=job.job_deliveryservice
) AS delivery_service,
job.id,
- job.keyword,
- job.parameters,
- job.start_time
+ ttl_hr,
+ start_time,
+ invalidation_type
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -130,11 +156,35 @@ WHERE server.status NOT IN (
const updateQuery = `
UPDATE job
SET asset_url=$1,
- keyword=$2,
- parameters=$3,
- start_time=$4
+ ttl_hr=$2,
+ start_time=$3
+WHERE job.id=$4
+RETURNING asset_url,
+ (
+ SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS delivery_service,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ start_time
+`
+
+// Almost the same as updateQuery, but returns appropriate values for API 4.0+
+const updateQueryV40 = `
Review comment:
We should probably call this V4 instead of V40, because we'll likely just update this query with new fields if they're ever added.
##########
File path: traffic_ops/v4-client/job.go
##########
@@ -28,7 +28,7 @@ import (
const apiJobs = "/jobs"
// CreateInvalidationJob creates the passed Content Invalidation Job.
-func (to *Session) CreateInvalidationJob(job tc.InvalidationJobInput, opts RequestOptions) (tc.Alerts, toclientlib.ReqInf, error) {
+func (to *Session) CreateInvalidationJob(job tc.InvalidationJobCreateV40, opts RequestOptions) (tc.Alerts, toclientlib.ReqInf, error) {
Review comment:
The client methods should also use the `V4` alias as opposed to the `V40` struct type.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -168,11 +238,35 @@ INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
WHERE job.id=$1
`
+// Deprecated, only to be used with versions below 4.0
const deleteQuery = `
DELETE
FROM job
WHERE job.id=$1
RETURNING job.asset_url,
+ ( SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS deliveryservice,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time
+`
+
+// Almost the same as deleteQuery, but returns appropriate values for API 4.0+
+const deleteQueryV40 = `
+DELETE
+FROM job
+WHERE job.id=$1
+RETURNING
+ job.id,
+ job.asset_url,
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -168,11 +238,35 @@ INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
WHERE job.id=$1
`
+// Deprecated, only to be used with versions below 4.0
const deleteQuery = `
DELETE
FROM job
WHERE job.id=$1
RETURNING job.asset_url,
+ ( SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS deliveryservice,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time
+`
+
+// Almost the same as deleteQuery, but returns appropriate values for API 4.0+
+const deleteQueryV40 = `
Review comment:
We should probably call this V4 instead of V40, because we'll likely just update this query with new fields if they're ever added.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -130,11 +156,35 @@ WHERE server.status NOT IN (
const updateQuery = `
UPDATE job
SET asset_url=$1,
- keyword=$2,
- parameters=$3,
- start_time=$4
+ ttl_hr=$2,
+ start_time=$3
+WHERE job.id=$4
+RETURNING asset_url,
+ (
+ SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS delivery_service,
+ job.id,
+ 'PURGE' as keyword,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ start_time
+`
+
+// Almost the same as updateQuery, but returns appropriate values for API 4.0+
+const updateQueryV40 = `
+UPDATE job
+SET asset_url=$1,
+ ttl_hr=$2,
+ start_time=$3,
+ invalidation_type=$4
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -168,11 +238,35 @@ INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
WHERE job.id=$1
`
+// Deprecated, only to be used with versions below 4.0
const deleteQuery = `
DELETE
FROM job
WHERE job.id=$1
RETURNING job.asset_url,
+ ( SELECT tm_user.username
+ FROM tm_user
+ WHERE tm_user.id=job.job_user
+ ) AS created_by,
+ (
+ SELECT deliveryservice.xml_id
+ FROM deliveryservice
+ WHERE deliveryservice.id=job.job_deliveryservice
+ ) AS deliveryservice,
+ job.id,
+ 'PURGE' as keyword,
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
+
+ accessibleTenants, err := tenant.GetUserTenantIDListTx(job.APIInfo().Tx.Tx, job.APIInfo().User.TenantID)
+ if err != nil {
+ return nil, nil, fmt.Errorf("getting accessible tenants for user - %v", err), http.StatusInternalServerError, nil
+ }
+ if len(where) > 0 {
+ where += " AND ds.tenant_id = ANY(:tenants) "
+ } else {
+ where = dbhelpers.BaseWhere + " ds.tenant_id = ANY(:tenants) "
+ }
+ queryValues["tenants"] = pq.Array(accessibleTenants)
+
+ if useIMS {
+ runSecond, maxTime = ims.TryIfModifiedSinceQuery(job.APIInfo().Tx, h, queryValues, selectMaxLastUpdatedQuery(where))
+ if !runSecond {
+ log.Debugln("IMS HIT")
+ return []interface{}{}, nil, nil, http.StatusNotModified, &maxTime
+ }
+ log.Debugln("IMS MISS")
+ } else {
+ log.Debugln("Non IMS request")
+ }
+
+ query := readQueryV40 + where + orderBy + pagination
+ log.Debugln("generated job query: " + query)
+ log.Debugf("executing with values: %++v\n", queryValues)
+
+ returnable := []interface{}{}
+ rows, err := job.APIInfo().Tx.NamedQuery(query, queryValues)
+ if err != nil {
+ return nil, nil, fmt.Errorf("querying: %v", err), http.StatusInternalServerError, nil
+ }
+ defer rows.Close()
+
+ for rows.Next() {
+ job := tc.InvalidationJobV40{}
+ err := rows.Scan(&job.ID,
+ &job.AssetURL,
+ &job.CreatedBy,
+ &job.DeliveryService,
+ &job.TTLHours,
+ &job.InvalidationType,
+ &job.StartTime)
+ if err != nil {
Review comment:
nit: this error check is normally combined into the Scan like this:
```
if err := rows.Scan(...); err != nil {
...
}
```
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -183,9 +277,8 @@ RETURNING job.asset_url,
FROM deliveryservice
WHERE deliveryservice.id=job.job_deliveryservice
) AS deliveryservice,
- job.id,
- job.keyword,
- job.parameters,
+ ttl_hr,
+ job.invalidation_type,
Review comment:
formatting is off
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -146,20 +196,40 @@ RETURNING job.asset_url,
WHERE deliveryservice.id=job.job_deliveryservice
) AS delivery_service,
job.id,
- job.keyword,
- job.parameters,
- job.start_time
+ ttl_hr,
+ start_time,
+ invalidation_type
`
+// Deprecated, only to be used with versions below 4.0
const putInfoQuery = `
SELECT job.id AS id,
tm_user.username AS createdBy,
job.job_user AS createdByID,
job.job_deliveryservice AS dsid,
deliveryservice.xml_id AS dsxmlid,
job.asset_url AS assetURL,
- job.parameters,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time AS start_time,
+ origin.protocol || '://' || origin.fqdn || rtrim(concat(':', origin.port), ':') AS OFQDN
+FROM job
+INNER JOIN origin ON origin.deliveryservice=job.job_deliveryservice AND origin.is_primary
+INNER JOIN tm_user ON tm_user.id=job.job_user
+INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
+WHERE job.id=$1
+`
+
+// Almost the same as putInfoQuery, but returns appropriate values for API 4.0+
+const putInfoQueryV40 = `
Review comment:
We should probably call this V4 instead of V40, because we'll likely just update this query with new fields if they're ever added.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -270,6 +475,12 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
return nil, nil, fmt.Errorf("parsing db response: %v", err), http.StatusInternalServerError, nil
}
+ // NamedQuery (nor prepare statements) cannot handle string functions in SELECT such as:
+ // CONCAT('TTL:', ttl_hr, 'h') AS parameters
Review comment:
I'm guessing this might be because of the `:` in `TTL:`. I think using `TTL::` might work? That is to say, I think it might actually be possible to use `CONCAT` and other string functions in `NamedQuery`.
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
Review comment:
We should probably call this V4 instead of V40, because we'll likely just update this query with new fields if they're ever added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727577725
##########
File path: lib/go-atscfg/atscfg.go
##########
@@ -655,12 +654,6 @@ func makeErr(warnings []string, err string) error {
return errors.New(`(warnings: ` + strings.Join(warnings, `, `) + `) ` + err)
}
-// makeErrf is a convenience for formatting errors for makeErr.
-// todo also unused, maybe remove?
-func makeErrf(warnings []string, format string, v ...interface{}) error {
Review comment:
I would prefer we didn't remove this. It isn't used at the moment, but it's generally useful as a library function.
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -57,7 +56,7 @@ func MakeRegexRevalidateDotConfig(
server *Server,
deliveryServices []DeliveryService,
globalParams []tc.Parameter,
- jobs []tc.InvalidationJob,
+ jobs []tc.InvalidationJobV4,
Review comment:
Please don't add versioned structs to `lib/go-atscfg`. The `lib/go-tc` library changes and breaks so much, they're extremely painful to deal with.
The best solution we've found to minimize the pain, is to make type aliases, which can be updated and then only the broken things need fixed, instead of every single reference.
https://github.com/apache/trafficcontrol/blob/master/lib/go-atscfg/atscfg.go#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-958029173
TP integration test failure appears to be a transient issue unrelated to this PR, merging ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r715835370
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
> In fact, we probably already have a function in `dbhelpers.go` that does exactly that.
Looks like we don't, actually. Best you could do looks like using `GetDSTenantIDFromXMLID` and just throwing away the tenant ID return value. Better solution would be to make a simple `DSExists` function in that package, IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r706361680
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
Normally we like to hide the specific _minor_ version and instead only reference the _major_ version everywhere that's external to the API implementation (and even then, the specific minor version references should be as limited as possible). Typically like this:
```
type InvalidationJobV4 InvalidationJobV40 // this alia should always point to the latest minor version
```
So this test would instead reference the `InvalidationJobV4` alias type.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
Review comment:
this should be `InvalidationJobsResponseV4` and contain `[]InvalidationJobV4`
##########
File path: lib/go-util/ptr.go
##########
@@ -1,5 +1,7 @@
package util
+import "time"
Review comment:
imports typically come after the Apache header
##########
File path: cache-config/t3cutil/getdatacfg.go
##########
@@ -69,7 +69,7 @@ type ConfigData struct {
Server *atscfg.Server `json:"server,omitempty"`
// Jobs must be all Jobs on the server's CDN. May include jobs on other CDNs.
- Jobs []tc.InvalidationJob `json:"jobs,omitempty"`
+ Jobs []tc.InvalidationJobV40 `json:"jobs,omitempty"`
Review comment:
Per my earlier comment, this would be `InvalidationJobV4`
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -232,3 +202,22 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
return newJobs, warnings
}
+
+const MISS = "MISS"
+const STALE = "STALE"
+
+// ProcessRefetch determines the type of Invalidation, returns the corresponding jobtype
+// and "cleans" the regex URL for the asset to be invalidated. REFETCH trumps REFRESH,
+// whether in the AssetURL or as InvalidationType
+func ProcessRefetch(invalidationType, assetURL string) (string, string) {
Review comment:
it seems like this function should be private
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -197,25 +179,13 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
if tcJob.StartTime.Add(ttl).Before(time.Now()) {
continue
}
- if tcJob.Keyword == nil || *tcJob.Keyword != JobKeywordPurge {
- continue
- }
-
if tcJob.AssetURL == nil {
continue
}
// process the __REFETCH__ keyword
Review comment:
Is this even the keyword? I thought it used `##` instead of `__` -- should we just remove this comment?
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -197,25 +179,13 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
if tcJob.StartTime.Add(ttl).Before(time.Now()) {
continue
}
- if tcJob.Keyword == nil || *tcJob.Keyword != JobKeywordPurge {
- continue
- }
-
if tcJob.AssetURL == nil {
continue
}
// process the __REFETCH__ keyword
- assetURL := *tcJob.AssetURL
- var jobType string
-
- if strings.HasSuffix(assetURL, RefetchSuffix) {
- assetURL = strings.TrimSuffix(assetURL, RefetchSuffix)
- jobType = "MISS"
- } else if strings.HasSuffix(assetURL, RefreshSuffix) { // also default
- assetURL = strings.TrimSuffix(assetURL, RefreshSuffix)
- jobType = "STALE"
- }
+ // There is no InvalidationType string before TO v4.0, so pass empty string
Review comment:
this particular comment doesn't seem to match what is actually done
##########
File path: cache-config/t3cutil/toreq/clientfuncs.go
##########
@@ -513,24 +513,24 @@ func (cl *TOClient) GetDeliveryServiceRegexes(reqHdr http.Header) ([]tc.Delivery
return regexes, reqInf, nil
}
-func (cl *TOClient) GetJobs(reqHdr http.Header) ([]tc.InvalidationJob, toclientlib.ReqInf, error) {
+func (cl *TOClient) GetJobs(reqHdr http.Header) ([]tc.InvalidationJobV40, toclientlib.ReqInf, error) {
Review comment:
Per my earlier comment, all these references in this file would be `InvalidationJobV4`
##########
File path: lib/go-atscfg/regexrevalidatedotconfig_test.go
##########
@@ -41,46 +41,66 @@ func TestMakeRegexRevalidateDotConfig(t *testing.T) {
dses := []DeliveryService{*ds}
params := makeParamsFromMapArr("GLOBAL", RegexRevalidateFileName, map[string][]string{
- RegexRevalidateMaxRevalDurationDaysParamName: []string{"42"},
- "unrelated": []string{"unrelated0", "unrelated1"},
+ RegexRevalidateMaxRevalDurationDaysParamName: {"42"},
+ "unrelated": {"unrelated0", "unrelated1"},
})
- jobs := []tc.InvalidationJob{
- tc.InvalidationJob{
- AssetURL: util.StrPtr("assetURL0"),
- StartTime: &tc.Time{Time: time.Now().Add(42*24*time.Hour + time.Hour), Valid: true},
- DeliveryService: util.StrPtr("myds"),
- CreatedBy: util.StrPtr("me"),
- ID: util.UInt64Ptr(42),
- Parameters: util.StrPtr("TTL:14h"),
- Keyword: util.StrPtr(JobKeywordPurge),
+ jobs := []tc.InvalidationJobV40{
Review comment:
This file should also reference an `InvalidationJobV4` alias.
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -57,7 +57,7 @@ func MakeRegexRevalidateDotConfig(
server *Server,
deliveryServices []DeliveryService,
globalParams []tc.Parameter,
- jobs []tc.InvalidationJob,
+ jobs []tc.InvalidationJobV40,
Review comment:
this file should use the `InvalidationJobV4` alias as well
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
Review comment:
We should also alias this struct to `InvalidationJobCreateV4` and use the alias in the client code instead of `V40`
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -246,6 +246,12 @@ func (job *InvalidationJobInput) Validate(tx *sql.Tx) error {
return nil
}
+type compareJob struct {
+ AssetURL *string
Review comment:
If these DB fields are non-null (and I think that's true), do these have to be pointers?
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
Review comment:
This method should belong to the `InvalidationJobCreateV4` alias -- we wouldn't want to have to duplicate this and move it to a new struct if we added a minor version (4.1)
##########
File path: lib/go-atscfg/atscfg.go
##########
@@ -696,3 +696,35 @@ func JobToInvalidationJob(jb tc.Job) (tc.InvalidationJob, error) {
StartTime: &startTime,
}, nil
}
+
+func JobsToInvalidationJobsV40(oldJobs []tc.Job) ([]tc.InvalidationJobV40, error) {
Review comment:
This new code block should use the `InvalidationJobV4` alias as well.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
This really shouldn't be a method on this struct at all, it should just be a simple function that takes an xml_id as input and is used in both the pre-4.0 and 4.0 APIs.
In fact, we probably already have a function in `dbhelpers.go` that does exactly that. All of this validation code really belongs in traffic_ops_golang IMO, in which we could make use of those pre-existing functions. That would also remove unnecessary references to `sql` in this library.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
Review comment:
this method should belong to `InvalidationJobV4`
Also, to save myself from commenting this on every method, all these methods in the rest of this addition should belong to `InvalidationJobV4`. The idea being that, if we add a new minor version, we just have to update the alias to point to the new minor version struct, and we don't also have to change all these function signatures all over the place.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
Review comment:
Above this we should alias this struct like so:
```
type InvalidationJobV4 InvalidationJobV40 // this alias should always point to the latest minor version
```
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -486,7 +491,7 @@ func MakeFakeTOData() *t3cutil.ConfigData {
},
DeliveryServiceServers: dss,
Server: sv0,
- Jobs: []tc.InvalidationJob{
+ Jobs: []tc.InvalidationJobV40{
Review comment:
Per my prior comment, this would be `InvalidationJobV4`
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
Review comment:
If this were in `traffic_ops_golang`, we could log the actual error and return this more redacted error to the user.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
Review comment:
Instead of silently ignoring, it's always better to explicitly ignore via `_`. Also, if this is moved into `traffic_ops_golang`, we could log the error if one occurs.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
If this were in `traffic_ops_golang`, we could log the actual error and return this more redacted error to the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725186807
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -197,25 +179,13 @@ func filterJobs(tcJobs []tc.InvalidationJob, maxReval time.Duration, minTTL time
if tcJob.StartTime.Add(ttl).Before(time.Now()) {
continue
}
- if tcJob.Keyword == nil || *tcJob.Keyword != JobKeywordPurge {
- continue
- }
-
if tcJob.AssetURL == nil {
continue
}
// process the __REFETCH__ keyword
Review comment:
Agreed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727579240
##########
File path: lib/go-atscfg/regexrevalidatedotconfig.go
##########
@@ -57,7 +56,7 @@ func MakeRegexRevalidateDotConfig(
server *Server,
deliveryServices []DeliveryService,
globalParams []tc.Parameter,
- jobs []tc.InvalidationJob,
+ jobs []tc.InvalidationJobV4,
Review comment:
Please don't add versioned structs to `lib/go-atscfg`. The `lib/go-tc` library changes and breaks so much, they're extremely painful to deal with.
The best solution we've found to minimize the pain, is to make type aliases, which can be updated and then only the broken things need fixed, instead of every single reference.
https://github.com/apache/trafficcontrol/blob/master/lib/go-atscfg/atscfg.go#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726609411
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
We haven't always been the best at sticking to major-version only while hiding most minor-version details. In general, just think about the amount of code that needs to change when we inevitable add another minor version. You want the blast-radius to be as small as possible when that happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727569574
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
Included the changes and tests to verify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724634848
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -42,52 +42,40 @@ import (
"github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/tenant"
)
+// Deprecated, only to be used with versions below 4.0
type InvalidationJob struct {
api.APIInfoImpl `json:"-"`
tc.InvalidationJob
}
-const readQuery = `
-SELECT job.id,
- keyword,
- parameters,
- asset_url,
- start_time,
- u.username AS createdBy,
- ds.xml_id AS dsId
-FROM job
-JOIN tm_user u ON job.job_user = u.id
-JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
-`
+type InvalidationJobV40 struct {
+ api.APIInfoImpl `json:"-"`
+ tc.InvalidationJobV40
+}
Review comment:
CRUDer is fine if that's how you wanna do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725314788
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
Gotcha. I can move business and validation logic into `traffic_ops_golang`, but keep the struct definitions in `lib/go-tc` to be shared between client and server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724574893
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
Good points! Should I move this, then, into _traffic\_ops\_golang_? I was mirroring the implementation of the previous `func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error` function, but extracting the db calls into separate funcs. If I move the db calls into, say, `dbhelpers` then I think I would be stuck as to whether the logic should reside in the the _/lib/go-tc_ or in _traffic\_ops\_golang_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725312610
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -146,20 +196,40 @@ RETURNING job.asset_url,
WHERE deliveryservice.id=job.job_deliveryservice
) AS delivery_service,
job.id,
- job.keyword,
- job.parameters,
- job.start_time
+ ttl_hr,
+ start_time,
+ invalidation_type
`
+// Deprecated, only to be used with versions below 4.0
const putInfoQuery = `
SELECT job.id AS id,
tm_user.username AS createdBy,
job.job_user AS createdByID,
job.job_deliveryservice AS dsid,
deliveryservice.xml_id AS dsxmlid,
job.asset_url AS assetURL,
- job.parameters,
+ CONCAT('TTL:', ttl_hr, 'h') AS parameters,
+ job.start_time AS start_time,
+ origin.protocol || '://' || origin.fqdn || rtrim(concat(':', origin.port), ':') AS OFQDN
+FROM job
+INNER JOIN origin ON origin.deliveryservice=job.job_deliveryservice AND origin.is_primary
+INNER JOIN tm_user ON tm_user.id=job.job_user
+INNER JOIN deliveryservice ON deliveryservice.id=job.job_deliveryservice
+WHERE job.id=$1
+`
+
+// Almost the same as putInfoQuery, but returns appropriate values for API 4.0+
+const putInfoQueryV40 = `
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724635561
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
+ }
+ return dsID, nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+// TODO: Error is swallowed from DB...
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
+ tx.QueryRow(`SELECT value FROM parameter WHERE name='refetch_enabled' AND config_file='global'`).Scan(&refetchEnabled)
Review comment:
For whatever reason I didn't see these before, so it may be too late at this point, but what I was talking about was not using a Parameter at all, instead using a property in Traffic Ops's configuration file (typically named `cdn.config` or `cdn.conf` or `cdn.json`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725326759
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r726614574
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
> Though, to add to this, aren't the status values just as relative to whomever is managing their CDN system? Or do we require that a job status of PENDING exist in all installations of TO/ATC?
The `PENDING` status is created via seeds.sql, so we do require that job status to exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725341990
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Excellent point and definitely correct over hardcoded `1`. Though, to add to this, aren't the status values just as relative to whomever is managing their CDN system? Or do we require that a job status of `PENDING` exist in all installations of TO/ATC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725328714
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
From an earlier answer, will move this particular functionality into `dbhelpers` as part of moving the validation logic itself into `traffic_ops_golang`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r725322719
##########
File path: lib/go-atscfg/regexrevalidatedotconfig_test.go
##########
@@ -41,46 +41,66 @@ func TestMakeRegexRevalidateDotConfig(t *testing.T) {
dses := []DeliveryService{*ds}
params := makeParamsFromMapArr("GLOBAL", RegexRevalidateFileName, map[string][]string{
- RegexRevalidateMaxRevalDurationDaysParamName: []string{"42"},
- "unrelated": []string{"unrelated0", "unrelated1"},
+ RegexRevalidateMaxRevalDurationDaysParamName: {"42"},
+ "unrelated": {"unrelated0", "unrelated1"},
})
- jobs := []tc.InvalidationJob{
- tc.InvalidationJob{
- AssetURL: util.StrPtr("assetURL0"),
- StartTime: &tc.Time{Time: time.Now().Add(42*24*time.Hour + time.Hour), Valid: true},
- DeliveryService: util.StrPtr("myds"),
- CreatedBy: util.StrPtr("me"),
- ID: util.UInt64Ptr(42),
- Parameters: util.StrPtr("TTL:14h"),
- Keyword: util.StrPtr(JobKeywordPurge),
+ jobs := []tc.InvalidationJobV40{
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp merged pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp merged pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724578710
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +463,210 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+func (job *InvalidationJobCreateV40) GetDSIDfromDSXMLID(tx *sql.Tx) (uint, error) {
+ var dsID uint
+ row := tx.QueryRow(`SELECT id FROM deliveryservice WHERE xml_id=$1`, job.DeliveryService)
+ if err := row.Scan(&dsID); err != nil {
+ if err == sql.ErrNoRows {
+ return 0, fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return 0, errors.New("Unknown error occurred")
Review comment:
The logic should also go into `traffic_ops_golang` IMO. Basically, `lib/go-tc` was only meant to be for the various things that needed to be shared among components -- primarily TO and the TO client (for the data structs). Over time, things like validation logic have snuck into `lib/go-tc`, but really validation logic belongs in `traffic_ops_golang` IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r724590596
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -282,6 +493,152 @@ func (job *InvalidationJob) Read(h http.Header, useIMS bool) ([]interface{}, err
// Used by POST requests to `/jobs`, creates a new content invalidation job
// from the provided request body.
+func CreateV40(w http.ResponseWriter, r *http.Request) {
+ inf, userErr, sysErr, errCode := api.NewInfo(r, nil, nil)
+ if userErr != nil || sysErr != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, errCode, userErr, sysErr)
+ return
+ }
+ defer inf.Close()
+
+ job := tc.InvalidationJobCreateV40{}
+ if err := json.NewDecoder(r.Body).Decode(&job); err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusBadRequest, errors.New("Unable to parse Invalidation Job"), fmt.Errorf("parsing jobs/ POST: %v", err))
+ return
+ }
+
+ // Check if request object is valid
+ w.Header().Set(rfc.ContentType, rfc.ApplicationJSON)
+ if err := job.Validate(inf.Tx.Tx); err != nil {
+ response := tc.Alerts{
+ Alerts: []tc.Alert{
+ {
+ Text: err.Error(),
+ Level: tc.ErrorLevel.String(),
+ },
+ },
+ }
+
+ resp, err := json.Marshal(response)
+ if err != nil {
+ api.HandleErr(w, r, inf.Tx.Tx, http.StatusInternalServerError, nil, fmt.Errorf("Encoding bad request response: %v", err))
+ return
+ }
+
+ w.WriteHeader(http.StatusBadRequest)
+ api.WriteAndLogErr(w, r, append(resp, '\n'))
Review comment:
Will change!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727569964
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
Review comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#discussion_r727431924
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -183,9 +277,8 @@ RETURNING job.asset_url,
FROM deliveryservice
WHERE deliveryservice.id=job.job_deliveryservice
) AS deliveryservice,
- job.id,
- job.keyword,
- job.parameters,
+ ttl_hr,
+ job.invalidation_type,
Review comment:
Addressed
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
Review comment:
Addressed
##########
File path: traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
##########
@@ -203,17 +301,124 @@ func selectMaxLastUpdatedQuery(where string) string {
select max(last_updated) as t from last_deleted l where l.table_name='job') as res`
}
+// Deprecated, only to be used with versions below 4.0
+const readQuery = `
+SELECT job.id,
+ 'PURGE' AS keyword,
+ ttl_hr,
+ asset_url,
+ start_time,
+ u.username as createdBy,
+ ds.xml_id as dsId
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Almost the same as readQuery, but returns appropriate values for API 4.0+
+const readQueryV40 = `
+SELECT job.id,
+ asset_url,
+ u.username as createdBy,
+ ds.xml_id,
+ ttl_hr,
+ invalidation_type,
+ start_time
+FROM job
+JOIN tm_user u ON job.job_user = u.id
+JOIN deliveryservice ds ON job.job_deliveryservice = ds.id
+`
+
+// Used by GET requests to `/jobs`, simply returns a filtered list of
+// content invalidation jobs according to the provided query parameters.
+func (job *InvalidationJobV40) Read(h http.Header, useIMS bool) ([]interface{}, error, error, int, *time.Time) {
+ var maxTime time.Time
+ var runSecond bool
+ queryParamsToSQLCols := map[string]dbhelpers.WhereColumnInfo{
+ "id": dbhelpers.WhereColumnInfo{Column: "job.id", Checker: api.IsInt},
+ "assetUrl": dbhelpers.WhereColumnInfo{Column: "asset_url"},
+ "startTime": dbhelpers.WhereColumnInfo{Column: "start_time"},
+ "userId": dbhelpers.WhereColumnInfo{Column: "job_user", Checker: api.IsInt},
+ "createdBy": dbhelpers.WhereColumnInfo{Column: `(SELECT tm_user.username FROM tm_user WHERE tm_user.id=job.job_user)`},
+ "deliveryService": dbhelpers.WhereColumnInfo{Column: `(SELECT deliveryservice.xml_id FROM deliveryservice WHERE deliveryservice.id=job.job_deliveryservice)`},
+ "dsId": dbhelpers.WhereColumnInfo{Column: "job.job_deliveryservice", Checker: api.IsInt},
+ "invalidationType": dbhelpers.WhereColumnInfo{Column: "invalidation_type"},
+ }
+
+ where, orderBy, pagination, queryValues, errs := dbhelpers.BuildWhereAndOrderByAndPagination(job.APIInfo().Params, queryParamsToSQLCols)
+ if len(errs) > 0 {
+ return nil, util.JoinErrs(errs), nil, http.StatusBadRequest, nil
+ }
Review comment:
Included the changes and tests to verify.
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Done
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
Review comment:
Moved
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
+ var exists bool
+ row := tx.QueryRow(`SELECT EXISTS(SELECT * FROM deliveryservice WHERE xml_id=$1)`, job.DeliveryService)
+ if err := row.Scan(&exists); err != nil {
+ if err == sql.ErrNoRows {
+ return fmt.Errorf("No DeliveryService exists matching identifier: %v", job.DeliveryService)
+ }
+ return errors.New("Unknown error occurred")
+ }
+ return nil
+}
+
+// validateTLLHours ensures the supplied TTL hours is within acceptable limits
+func (job *InvalidationJobCreateV40) validateTLLHours(tx *sql.Tx) error {
+ var maxDays uint
+ err := tx.QueryRow(`SELECT value FROM parameter WHERE name='maxRevalDurationDays' AND config_file='regex_revalidate.config'`).Scan(&maxDays)
+ maxHours := maxDays * 24
+ if err == nil && uint(job.TTLHours) > maxHours { // silently ignore other errors too
+ return fmt.Errorf("cannot exceed %s", strconv.FormatUint(uint64(maxHours), 10))
+ }
+ return nil
+}
+
+// RefetchAllowed checks whether Refetch is allowed and enabled as a global paramter
+func RefetchAllowed(tx *sql.Tx) bool {
+ refetchEnabled := false
+ // Silently ignore paramter error
Review comment:
Moved into `traffic_ops_golang`. Now log the error.
##########
File path: lib/go-tc/invalidationjobs.go
##########
@@ -456,3 +462,209 @@ func (job *UserInvalidationJobInput) Validate(tx *sql.Tx) error {
}
return nil
}
+
+const REFRESH = "REFRESH"
+const REFETCH = "REFETCH"
+
+// InvalidationJobsResponse is the type of a response from Traffic Ops to a
+// request made to its /jobs API endpoint for v 4.0+
+type InvalidationJobsResponseV40 struct {
+ Response []InvalidationJobV40 `json:"response"`
+ Alerts
+}
+
+// InvalidationJobCreateV40 represents user input intending to create a content invalidation job.
+type InvalidationJobCreateV40 struct {
+ // The Delivery Service XML-ID for which the Invalidation Job is to be applied.
+ DeliveryService string `json:"deliveryService"`
+
+ // Regex is a regular expression which not only must be valid, but should also start with '/'
+ // (or escaped: '\/')
+ Regex string `json:"regex"`
+
+ // StartTime is the time at which the job will come into effect. Must be in the future.
+ StartTime time.Time `json:"startTime"`
+
+ // TTLHours indicates the Time-to-Live of the job in hours. Must be a positive integer value.
+ TTLHours uint32 `json:"ttlHours"`
+
+ // InvalidationType must be either REFRESH (default behavior) or REFETCH. If REFETCH, must
+ // also comply with global parameter setting
+ InvalidationType string `json:"invalidationType"`
+}
+
+// Validates the fields submitted for an InvalidationJobCreateV40. These errors
+// are ultimately returned to the user
+func (job *InvalidationJobCreateV40) Validate(tx *sql.Tx) error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.Regex, validation.Required, validation.NewStringRule(func(s string) bool {
+ return strings.HasPrefix(s, `\/`) || strings.HasPrefix(s, "/")
+ }, `must start with '/' (or '\/')`)),
+ validation.Field(&job.StartTime, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if err := job.validateDeliveryService(tx); err != nil {
+ errs = append(errs, "Delivery Service is invalid: "+err.Error())
+ }
+
+ if _, err := regexp.Compile(job.Regex); err != nil {
+ errs = append(errs, "regex: is not a valid Regular Expression: "+err.Error())
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: must be in the future")
+ }
+
+ if err := job.validateTLLHours(tx); err != nil {
+ errs = append(errs, "TTL is invalid: "+err.Error())
+ }
+
+ if job.InvalidationType == REFETCH && !RefetchAllowed(tx) {
+ errs = append(errs, "InvalidationType is invalid")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// InvalidationJobV40 represents a content invalidation job as returned by the API.
+// Also used for Update calls.
+type InvalidationJobV40 struct {
+ ID *uint64 `json:"id"`
+ AssetURL *string `json:"assetUrl"`
+ CreatedBy *string `json:"createdBy"`
+ DeliveryService *string `json:"deliveryService"`
+ TTLHours *uint `json:"ttlHours"`
+ InvalidationType *string `json:"invalidationType"`
+ StartTime *time.Time `json:"startTime"`
+}
+
+func (job InvalidationJobV40) String() string {
+ var ID, AssetURL, CreatedBy, DeliveryService, TTLHours, InvalidationType, StartTime string
+ if job.ID != nil {
+ ID = strconv.FormatUint(*job.ID, 10)
+ } else {
+ ID = "-1"
+ }
+ if job.AssetURL != nil {
+ AssetURL = *job.AssetURL
+ }
+ if job.CreatedBy != nil {
+ CreatedBy = *job.CreatedBy
+ }
+ if job.DeliveryService != nil {
+ DeliveryService = *job.DeliveryService
+ }
+ if job.TTLHours != nil {
+ TTLHours = strconv.FormatUint(uint64(*job.TTLHours), 10)
+ }
+ if job.InvalidationType != nil {
+ InvalidationType = *job.InvalidationType
+ }
+ if job.StartTime != nil {
+ StartTime = job.StartTime.String()
+ }
+ return fmt.Sprintf("{\nID:\t\t\t\t\t%s,\nAssetURL:\t\t\t%s,\nCreatedBy:\t\t\t%s,\nDeliveryService:\t%s,\nTTLHours:\t\t\t%s,\nInvalidationType:\t%s,\nStartTime:\t\t\t%s,\n}",
+ ID,
+ AssetURL,
+ CreatedBy,
+ DeliveryService,
+ TTLHours,
+ InvalidationType,
+ StartTime)
+}
+
+// Validate checks that the InvalidationJob is valid, by ensuring all of its fields are well-defined.
+//
+// This returns an error describing any and all problematic fields encountered during validation.
+func (job *InvalidationJobV40) Validate() error {
+ errs := []string{}
+ err := validation.ValidateStruct(job,
+ validation.Field(&job.DeliveryService, validation.Required),
+ validation.Field(&job.AssetURL, validation.Required, is.URL),
+ validation.Field(&job.CreatedBy, validation.Required),
+ validation.Field(&job.ID, validation.Required),
+ validation.Field(&job.TTLHours, validation.Required),
+ validation.Field(&job.InvalidationType, validation.Required, validation.NewStringRule(func(s string) bool {
+ return s == REFRESH || s == REFETCH
+ }, fmt.Sprintf("must be either %s or %s (case sensitive)", REFRESH, REFETCH))),
+ )
+
+ if err != nil {
+ errs = append(errs, err.Error())
+ }
+
+ if job.StartTime == nil {
+ return errors.New(strings.Join(append(errs, "startTime: cannot be blank"), ", "))
+ }
+
+ if job.StartTime.After(time.Now().Add(twoDays)) {
+ errs = append(errs, "startTime: must be within two days from now")
+ }
+
+ if job.StartTime.Before(time.Now()) {
+ errs = append(errs, "startTime: cannot be in the past")
+ }
+
+ if len(errs) > 0 {
+ return errors.New(strings.Join(errs, ", "))
+ }
+
+ return nil
+}
+
+// validateDeliveryService ensures the supplied (required) Delivery Service XML ID exists
+func (job *InvalidationJobCreateV40) validateDeliveryService(tx *sql.Tx) error {
Review comment:
Moved
##########
File path: cache-config/t3c-generate/cfgfile/cfgfile_test.go
##########
@@ -351,16 +356,16 @@ func randParam() *tc.Parameter {
}
}
-func randJob() *tc.InvalidationJob {
- now := &tc.Time{Time: time.Now(), Valid: true}
- return &tc.InvalidationJob{
- Parameters: randStr(),
- Keyword: randStr(),
- AssetURL: randStr(),
- CreatedBy: randStr(),
- StartTime: now,
- ID: randUint64(),
- DeliveryService: randStr(),
+func randJob() *tc.InvalidationJobV40 {
Review comment:
Most excellent. Ty
##########
File path: traffic_ops/testing/api/v4/tc-fixtures.json
##########
@@ -5462,20 +5462,23 @@
{
"deliveryService": "ds1",
"regex": "/.*",
- "startTime": 4117118271000,
- "ttl": "121m"
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 72,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds1",
"regex": "/foo",
- "startTime": 4117118271000,
- "ttl": 2160
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 36,
+ "invalidationType" : "REFRESH"
},
{
"deliveryService": "ds2",
"regex": "\\/some-path?.+\\.jpg",
- "startTime": "2100-06-19T13:57:51-06:00",
- "ttl": 2.1
+ "startTime": "2100-06-19T13:57:51Z",
+ "ttlHours": 144,
+ "invalidationType" : "REFRESH"
Review comment:
I agree we need to add some `REFETCH` jobs. The order of operations with requiring a parameter value makes it a little difficult to add to the tc-fixtures file though. I will work on adding these in the go test file explicitly.
##########
File path: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+-- Revert
+
+/**
+ * Restore job_agent table with constraints and data.
+ */
+
+CREATE TABLE public.job_agent (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ active int4 NOT NULL DEFAULT 0,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_agent_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_agent_name_unique UNIQUE (name)
+);
+CREATE INDEX job_agent_last_updated_idx ON public.job_agent USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_agent for each row execute function on_delete_current_timestamp_last_updated('job_agent');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_agent for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_agent OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_agent TO traffic_ops;
+
+-- Data
+
+INSERT INTO public.job_agent (name, description, active)
+VALUES ('Example', 'Description of Purge Agent', 1)
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job_status table with constraints and data
+ */
+
+CREATE TABLE public.job_status (
+ id bigserial NOT NULL,
+ "name" text NULL,
+ description text NULL,
+ last_updated timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT idx_job_status_id_primary PRIMARY KEY (id),
+ CONSTRAINT job_status_name_unique UNIQUE (name)
+);
+CREATE INDEX job_status_last_updated_idx ON public.job_status USING btree (last_updated DESC NULLS LAST);
+
+-- Table Triggers
+
+create trigger on_delete_current_timestamp after
+delete
+ on
+ public.job_status for each row execute function on_delete_current_timestamp_last_updated('job_status');
+create trigger on_update_current_timestamp before
+update
+ on
+ public.job_status for each row execute function on_update_current_timestamp_last_updated();
+
+-- Permissions
+
+ALTER TABLE public.job_status OWNER TO traffic_ops;
+GRANT ALL ON TABLE public.job_status TO traffic_ops;
+
+-- Data
+
+INSERT INTO job_status (name, description)
+VALUES ('PENDING', 'Job is queued, but has not been picked up by any agents yet'),
+('IN_PROGRESS', 'Job is being processed by agents'),
+('COMPLETED', 'Job has finished'),
+('CANCELLED', 'Job was cancelled')
+ON CONFLICT (name) DO NOTHING;
+
+/**
+ * Restore job table with constraints and data
+ */
+
+-- Restore table
+ALTER TABLE public.job
+ADD COLUMN IF NOT EXISTS agent int8 NULL,
+ADD COLUMN IF NOT EXISTS object_type text NULL,
+ADD COLUMN IF NOT EXISTS object_name text NULL,
+ADD COLUMN IF NOT EXISTS keyword text NULL,
+ADD COLUMN IF NOT EXISTS asset_type text NULL,
+ADD COLUMN IF NOT EXISTS status int8 NULL;
+
+ALTER TABLE public.job
+DROP COLUMN IF EXISTS invalidation_type CASCADE;
+
+-- Restore indices and foreign key constraints
+CREATE INDEX idx_job_fk_job_agent_id ON public.job USING btree (agent);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_agent_id FOREIGN KEY (agent) REFERENCES public.job_agent(id) ON DELETE CASCADE;
+
+CREATE INDEX idx_job_fk_job_status_id ON public.job USING btree (status);
+ALTER TABLE public.job ADD CONSTRAINT job_fk_job_status_id FOREIGN KEY (status) REFERENCES public.job_status(id);
+
+-- Restore data
+UPDATE public.job
+SET agent = 1;
+
+UPDATE public.job
+SET status = 1;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on pull request #6118: Feature/refetch
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on pull request #6118:
URL: https://github.com/apache/trafficcontrol/pull/6118#issuecomment-903052933
I guess it's not fair to expect him to do more work than he needs to. It's just so terrible right now I can't believe he thought any of it was worth keeping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org