You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2021/06/03 02:58:16 UTC

[GitHub] [trafficcontrol] TaylorCFrey opened a new pull request #5910: Blueprint/refetch

TaylorCFrey opened a new pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910


   
   ## What does this PR (Pull Request) do?
   
   This is a blueprint proposal to solve a few issues, however the primary one is:
   - [X] Is related to #5825 
   
   ## Which Traffic Control components are affected by this PR?
   
   This blueprint has the potential to impact:
   
   - CDN in a Box
   - Documentation
   - Traffic Control Clients (Go)
   - Traffic Ops
   - Traffic Ops ORT
   - Traffic Portal
   - CI tests
   
   ## What is the best way to verify this PR?
   
   No verification necessary for the blueprint. Just review the file `blueprints/refetch_invalidation.md`
   
   <!--
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645795971



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       Like if you ran `go fmt` on a file that included that code, it would change it to:
   ```go
   type InvalidationJobInput {
   	DeliveryService *interface{} `json:"deliveryService"`
   	Regex           *string      `json:"regex"`
   	StartTime       *Time        `json:"startTime"`
   	TTL             *interface{} `json:"ttl"`
   	dsid            *uint
   	ttl             *time.Duration
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651123605



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -98,13 +109,44 @@ Since the field is optional and existing functionality only signifies a **REFRES
 The response will be modified, then, to return this new value as well.
 
 Sample current response:
-```
-{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"

Review comment:
       Good catch, fixed. Also added information stating any Time fields will be RFC3339 compliant. However, there is a `jobs.go` file in `lib/go-tc` that is already deprecated that is using it's own time format.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651151327



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.

Review comment:
       I went ahead and made it explicit, but we can always opt for it to be optional if we decide to pivot going forward.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645264004



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.

Review comment:
       > is never a anything
   
   can be reworded




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rob05c commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648708673



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       We need to not allow creating those jobs at all, if the feature-flag isn't set. Because by the time it gets to the cache, it's too late: the user put in jobs, that they expected to work, and now we have an error we can't reasonably propagate up to them that the job won't work. 
   
   But we already knew it wasn't going to work, we have the flag in TO. We should be able to return an error from the /jobs POST.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648523923



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Could also be in the TO configuration file




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rob05c commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648711364



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Bearing in mind, these won't be intermittent errors. The endpoint will always return an error if the flag isn't set.
   
   That's a good reason to put the flag in the Parameter instead of the config file: the Param is already available in the API. Which lets UIs like the Portal fetch it, to decide whether to hide that extra button from users, so they never even see the option that will always error.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rawlinp merged pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rawlinp merged pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645877509



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Well, when I need an ATS and/or ORT/t3c/atstccfg expert I ask @rob05c - lets see if he has an opinion




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645792624



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       I agree, any jobs that don't have a numeric TTL parameter will fail to operate properly; I think a migration could reasonably parse it and change it to a numeric TTL column, since if the migration fails it means ORT is failing in some regard.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645789291



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Seems like there are two options when `MISS` is unavailable based on ATS version: ignore it or use refresh. Can you suggest which would be better?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645250606



##########
File path: traffic_portal/test/integration/README.md
##########
@@ -1,3 +1,4 @@
+

Review comment:
       I believe this was addressed in a subsequent push. Please let me know if you're still seeing this change.

##########
File path: go.mod
##########
@@ -54,7 +54,6 @@ require (
 	github.com/miekg/dns v1.0.6-0.20180406150955-01d59357d468
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.1 // indirect
-	github.com/ogier/pflag v0.0.2-0.20201025181535-73e519546fc0

Review comment:
       I believe this was addressed in a subsequent push. Please let me know if you're still seeing this change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645264004



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.

Review comment:
       > is never a anything
   
   can be reworded




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651148077



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",

Review comment:
       I added some clarity to the alert message response below.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] limited commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
limited commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645829051



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       I'm not an ATS expert, so I can't make a solid suggestion here. 
   
   I think whatever happens probably needs to be bubbled up to the operator. You don't want to silently ignore or silently do a stale-refresh when you expect a Miss-Refetch. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651154729



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`

Review comment:
       Another good observation. What is actually optional? Any of the fields? We can make them all explicitly required at this time as well if you believe that's a benefit at this time.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651234518



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       That is the purpose for introducing the feature, but as I understand it the implementation will have the same effects on any requests made to it's parent for the affected content - regardless of whether or not it's an origin. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651147535



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       Hm, this is a really good point and observation. My understanding is that this all revolves entirely around misconfigured _origins_, and even more specifically around _origins_ we can't "fix". If it was an issue within the CDN topology, would we be able to manually intervene in the chain?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648679075



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Maybe I'm misunderstanding how this is supposed to work. I thought it'd be like: you make a new job, if it's using "refetch" then TO checks if the feature flag is allowed. If it isn't, you can't make that job, and any existing jobs are filtered out of GET requests for the /jobs endpoint.
   
   It sounds like you're saying you set the flag in TO, then ORT/t3c/atstccfg looks at it and determines based on that what to do with jobs that use "refetch" - which makes sense, because otherwise I suppose I don't know how `t3c` could "check and log an error"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rob05c commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645885626



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       It's definitely not supported on older versions. It's a new feature of the regex_revalidate plugin.
   
   1. Unfortunately, there's not an easy way to bubble it up. We don't really have a mechanism to send data from caches to TO, and even if we did, it'd be pretty unwieldy to collect that from every single cache.
   
   We do have the ATS version in a Parameter in TO, we could hard-code it. But someone might backport the plugin changeset to an older version, and then ATC just won't allow using it. So that's not a great solution.
   
   2. the "refetch" feature is **only** necessary or useful if the Origin server is fundamentally broken and completely violating the HTTP Spec. Without "refetch," ATS does an IMS request. So, the only reason it's ever needed is when a Tenant's Origin is responding with a 304 when the asset actually is modified. In complete violation of RFC 7234 and every HTTP Caching specification. But of course, we deal with horribly broken Origins every day. But I do think we should make that clear in the docs, and maybe the UI.
   
   3. To work correctly, every single Cache has to support it. Which is mostly fine, since CDN operators don't _usually_ operate with widely varying ATS versions. Some do, but I think the majority have a single version and maybe some newer Canaries.
   
   With those 3 things in mind, I think the best solution might be to make it a config Feature Flag in Traffic Ops/+Portal, with Documentation that they must have version X of regex_revalidate. As an additional safety, we should also make `t3c` check and log an error (which operators can then Alarm on).
   
   It's not as good as bubbling it up, but considering all those things, that's the best solution I'm seeing at the moment.
   

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       It's definitely not supported on older versions. It's a new feature of the regex_revalidate plugin.
   
   1. Unfortunately, there's not an easy way to bubble it up. We don't really have a mechanism to send data from caches to TO, and even if we did, it'd be pretty unwieldy to collect that from every single cache.
   
       We do have the ATS version in a Parameter in TO, we could hard-code it. But someone might backport the plugin changeset to an older version, and then ATC just won't allow using it. So that's not a great solution.
   
   2. the "refetch" feature is **only** necessary or useful if the Origin server is fundamentally broken and completely violating the HTTP Spec. Without "refetch," ATS does an IMS request. So, the only reason it's ever needed is when a Tenant's Origin is responding with a 304 when the asset actually is modified. In complete violation of RFC 7234 and every HTTP Caching specification. But of course, we deal with horribly broken Origins every day. But I do think we should make that clear in the docs, and maybe the UI.
   
   3. To work correctly, every single Cache has to support it. Which is mostly fine, since CDN operators don't _usually_ operate with widely varying ATS versions. Some do, but I think the majority have a single version and maybe some newer Canaries.
   
   With those 3 things in mind, I think the best solution might be to make it a config Feature Flag in Traffic Ops/+Portal, with Documentation that they must have version X of regex_revalidate. As an additional safety, we should also make `t3c` check and log an error (which operators can then Alarm on).
   
   It's not as good as bubbling it up, but considering all those things, that's the best solution I'm seeing at the moment.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645763917



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",

Review comment:
       How about changing the response text to "Invalidation request (refetch) created for..."




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645297415



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       You absolutely _can_ do that - I would suggest instead that the field not be optional, and the client creates objects exactly as they desire them to exist with no room for surprises.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651121730



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact

Review comment:
       I've added a bit more of a description. There isn't much more to the construction of T3C's `regexrevalidate.config` file. This is already implemented, we are allowing it to be explicitly controlled via ATC's APIs, interfaces, and clients.
   
   Essentially, it remains the same format, but with an optional third field for each non-comment entry.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] srijeet0406 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
srijeet0406 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645289948



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       I was thinking maybe we should have all the `json` tags indented at the same level. Something like this:
   `
   	AssetURL        *string `json:"assetUrl"`
   	CreatedBy       *string `json:"createdBy"`
   	DeliveryService *string `json:"deliveryService"`
   	ID              *uint64 `json:"id"`
   	Keyword         *string `json:"keyword"`
   	Parameters      *string `json:"parameters"`
   `




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645762215



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       tbh i think the parameters column was a poor design (there should have simply been a TTL column) and we should probably not follow that approach. imo i would suggest hijacking the keyword column and maybe renaming it to invalidation_type or simply type with 2 acceptable values: refresh and refetch




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651234518



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       That is the purpose for introducing the feature, but as I understand it the implementation will have the same effects on any requests made by a cache server to its parent for the affected content - regardless of whether or not it's an origin. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651125768



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:

Review comment:
       I've pivoted a bit on this original idea. My first attempt was to change as little as possible, reusing the `parameters` column. However, I believe we all agree that explicity and purposefully is a better approach. I've taken the approach of explicit columns (and datatypes). I also added a recommendation which I think we should implement that performs a great deal of clean up for the _jobs_ table. We are carrying a decent amount of future-proofing along with us that doesn't appear to be utilized anywhere.
   
   Definitely let me know if this newer proposal is lacking in clarity.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r644989456



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       should also syntax highlight as JSON

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -98,13 +109,44 @@ Since the field is optional and existing functionality only signifies a **REFRES
 The response will be modified, then, to return this new value as well.
 
 Sample current response:
-```
-{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"

Review comment:
       Invalid JSON here - also, can the start times be RFC3339 compliant?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       You absolutely _can_ do that - I would suggest instead that the field not be optional, and the client creates objects exactly as they desire them to exist with no room for surprises.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651123849



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.

Review comment:
       Nice one! Fixed, and hopefully clearer (more clear?).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645176789



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Oh excellent question. What are your thoughts? I suppose I would probably make this a bit more strict.
   
   To be explicitly correct, the caller would have to send either `refetch`, `refresh`, or nothing at all. By that I mean they would not include the `invalidationType` key at all (which would then default to `refresh`). If they did include the key and the value was an empty string, that would result in a 400. In fact, any other value would result in a 400. I think this is probably the best approach. The response can contain a helpful message to why it was rejected to assist any callers in the future.
   
   Should it be case *in*sensitive?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645172263



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```

Review comment:
       Good call!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r654008714



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`

Review comment:
       Cool info on the PATCH. That makes sense. I've gone ahead and made both `DeliveryService` and `Regex` nonoptional (required for brevity?).

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       I modified the original description to reference a CDN rather than a cache. Also added clarity that this could be a parent along the chain.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       I've added some language specifically stating the addition of a global parameter, defaulting to rejecting `refetch` POSTS until specfically set to allow.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645252322



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -98,13 +109,44 @@ Since the field is optional and existing functionality only signifies a **REFRES
 The response will be modified, then, to return this new value as well.
 
 Sample current response:
-```
-{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"

Review comment:
       Invalid JSON here - also, can the start times be RFC3339 compliant?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645758125



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:

Review comment:
       agree. stuffing lots of values in a column seems problematic going forward. can't you just reuse the `keyword` column of the job table? currently, keyword is PURGE for EVERY job. maybe keyword should be REFRESH and REFETCH instead and do a database migration to change all existing PURGE values to REFRESH?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648703807



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       > It sounds like you're saying you set the flag in TO, then ORT/t3c/atstccfg looks at it and determines based on that what to do with jobs that use "refetch" - which makes sense, because otherwise I suppose I don't know how t3c could "check and log an error"
   
   Yes, that is how `use_reval_pending` works currently and is how I imagine a feature flag for this feature would work as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] limited commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
limited commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645623472



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:

Review comment:
       Which table is this in? 
   
   Why not add a new column rather than storing multiple values in the same column?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       What considerations are made for TrafficServer version dependencies? 
   
   Is `MISS` supported by all ATS versions? If not, what happens when an operator sets a `REFETCH` job on a set of caches that do not support that invalidation type?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact

Review comment:
       Can you please describe how the construction of the `regexrevalidate.config` file will change given that this an external interface of Traffic Control?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651153999



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`

Review comment:
       This is a good catch, and I believe it should be documented at the very least. Same with the `TTL` below as well. I'm happy to add this as part of the refactor effort, especially since I expect more `-` than `+` :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651122556



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       I've modified this to be explicitly either `refresh` or `refetch`. All other submissions, including missing/omitted, will be reject as a client error (400 - Bad Request).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645762215



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       tbh i think the parameters column was a poor design (there should have simply been a TTL column) and we should probably not follow that approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645172135



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       Hm. How do you mean? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645760530



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       so i don't think the cache queries the "origin server" per se. instead it queries the "parent server" which in some cases will be a parent mid (if the cache is an edge) and the origin server (if the cache is a mid) and is simply the "parent server" when a topology is imployed by a delivery service (where there can be N tiers).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651239676



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.

Review comment:
       I'm going to resolve this then, since the principal concern of the original comment has been addressed; there's already a discussion about whether or not it should be optional elsewhere.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645756084



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       yeah, surprises are not good imo. so i guess 400 if invalidationType != refresh or refetch. and because you are adding this to a new major version of the api (4.0), you have the luxury of adding the new required field to the request...but tbh, if everyone likes the idea of a default value, i'm good with that too :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] srijeet0406 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
srijeet0406 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r644865012



##########
File path: go.mod
##########
@@ -54,7 +54,6 @@ require (
 	github.com/miekg/dns v1.0.6-0.20180406150955-01d59357d468
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.1 // indirect
-	github.com/ogier/pflag v0.0.2-0.20201025181535-73e519546fc0

Review comment:
       Since you're just adding a blue print file, should you be changing the `go.mod` and `go.sum`?

##########
File path: traffic_portal/test/integration/README.md
##########
@@ -1,3 +1,4 @@
+

Review comment:
       Extra blank line added

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {

Review comment:
       same comment about json tags here

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Just to be clear, if the value of `invalidationType` is malformed or of the wrong data type, it will result in a `400`, else, if it is missing, or any value other than `refetch` or `refresh` will always default to `refresh`, correct?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       Could this be displayed on multiple lines? Makes reading it a bit easier.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```

Review comment:
       same comment about multiple lines in both the above responses.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       The json tags dont seem to be aligned properly.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}

Review comment:
       same comment about multiple lines here

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       nit: `derive`

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       I was thinking maybe we should have all the `json` tags indented at the same level. Something like this:
   `
   	AssetURL        *string `json:"assetUrl"`
   	CreatedBy       *string `json:"createdBy"`
   	DeliveryService *string `json:"deliveryService"`
   	ID              *uint64 `json:"id"`
   	Keyword         *string `json:"keyword"`
   	Parameters      *string `json:"parameters"`
   `

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       I was thinking maybe we should have all the `json` tags indented at the same level. 

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Yeah I think if the user includes the `invalidationType` key and gives a blank string, that could probably be treated as `refresh` as well, but I'm new to invalidation jobs so maybe @ocket8888 would know better.
   The following four types should be accepted IMO:
   1.) `refresh`
   2.) `refetch`
   3.) blank string -> defaults to `refresh`
   4.) `invalidationType` key absent -> defaults to `refresh`
   Anything else is treated as a 400. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] srijeet0406 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
srijeet0406 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645289948



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       I was thinking maybe we should have all the `json` tags indented at the same level. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] srijeet0406 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
srijeet0406 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r644865012



##########
File path: go.mod
##########
@@ -54,7 +54,6 @@ require (
 	github.com/miekg/dns v1.0.6-0.20180406150955-01d59357d468
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.1 // indirect
-	github.com/ogier/pflag v0.0.2-0.20201025181535-73e519546fc0

Review comment:
       Since you're just adding a blue print file, should you be changing the `go.mod` and `go.sum`?

##########
File path: traffic_portal/test/integration/README.md
##########
@@ -1,3 +1,4 @@
+

Review comment:
       Extra blank line added

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {

Review comment:
       same comment about json tags here

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Just to be clear, if the value of `invalidationType` is malformed or of the wrong data type, it will result in a `400`, else, if it is missing, or any value other than `refetch` or `refresh` will always default to `refresh`, correct?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       Could this be displayed on multiple lines? Makes reading it a bit easier.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```

Review comment:
       same comment about multiple lines in both the above responses.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       The json tags dont seem to be aligned properly.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}

Review comment:
       same comment about multiple lines here

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       nit: `derive`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] srijeet0406 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
srijeet0406 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645291855



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Yeah I think if the user includes the `invalidationType` key and gives a blank string, that could probably be treated as `refresh` as well, but I'm new to invalidation jobs so maybe @ocket8888 would know better.
   The following four types should be accepted IMO:
   1.) `refresh`
   2.) `refetch`
   3.) blank string -> defaults to `refresh`
   4.) `invalidationType` key absent -> defaults to `refresh`
   Anything else is treated as a 400. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645760530



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.

Review comment:
       so i don't think the cache queries the "origin server" per se. instead it queries the "parent server" which in some cases will be a parent mid (if the cache is an edge) and the origin server (if the cache is a mid) and is simply the "parent server" when a topology is imployed by a delivery service. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rob05c commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rob05c commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648706299



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       When I said "feature flag" I was thinking the TO configuration file; but a global Parameter also works.
   
   > I thought it'd be like: you make a new job, if it's using "refetch" then TO checks if the feature flag is allowed. If it isn't, you can't make that job
   
   That's what I was thinking, but also having a check in `t3c-apply` as a fail-safe, if it ever does get a job and the installed regex_revalidate is too old to understand it, it would log an error (but probably still do the reval, because an IMS reval is better than nothing and doesn't hurt if they wanted a no-IMS reval). 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648461267



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.

Review comment:
       Yeah, I think nil = refresh, empty/malformed = invalid.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Just because we have a new major version we can add this to doesn't mean we should just make every new change a breaking change. I don't think making this a new required field adds enough value to warrant that since we can easily provide a default instead.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       Just for awareness @TaylorCFrey, a "feature flag" in TO usually means a parameter in the `GLOBAL` profile to enable the new feature -- similar to what we've done in the past w/ `use_tenancy` and what we currently do w/ `use_reval_pending`. That sounds like a good solution here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] mitchell852 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
mitchell852 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645756084



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       yeah, surprises are not good imo. so i guess 400 if invalidationType != refresh or refetch. and because you are adding this to a new major version of the api (4.0), you have the luxury of adding the new required field to the request.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645798557



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`

Review comment:
       This was a `*interface{}` in APIv3 to give a transition period between using Delivery Service ID (APIv2 and earlier) to XMLID (hopefully all versions moving forward) - so this could be a `string`. It's not really related to the changes we're making, but it's something that could be cleaned up since we're making structure changes anyway.
   
   That would also eliminate the need for the unexported `dsid` and the hoops you need to jump through for that.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`

Review comment:
       I don't really like optional properties, but also why `omitempty`? That won't actually be allowed to be empty in the database, right?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`

Review comment:
       `regex` isn't actually optional, right? Can this just be a `string` instead of a `*string`?

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`

Review comment:
       Does anyone really like being able to put in "duration strings"? I don't think that's as useful as I thought it was. JSON and Javascript numbers are IEEE double-precision floating point numbers, which means that as long as we don't need jobs to last longer than 290 years this could just be a `time.Duration` which is a duration in nanoseconds.
   
   That would also eliminate the need for the unexported `ttl` and the hoops you need to jump through for that.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.

Review comment:
       So if I pass in a type that's `demo1`, for example, that'll be treated as `refresh`? That worries me a bit.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651146140



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       I've modified the proposal to be more explicit, rather than the first attempt (which was to make the most minimal changes). Relish the feedback.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651156028



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`

Review comment:
       The API call implementations (whether go clients or TP), from what I can gather, are sending up JS numbers. We can certainly enforce a numeric data type here, and even if we stick with the hours format, it would still be clearer than empty interface.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651238643



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`

Review comment:
       I don't think any of the properties were optional, but it's easier to tell for strings than other things if a value was actually given. For example `"regex": ""` is just as invalid as `"regex": null` or a missing `"regex"` property, and Go's parser can't tell the difference between those cases for a `string` property. However, for something numeric, Go can't tell the difference between `0`, `null` and undefined values for an `int` property, but sometimes `0` is a valid value, so you have to use `*int`. It's a very annoying and pedantic balancing act you have to perform with the number of things you have to check for `nil` all the time versus what the Go parser can and can't distinguish between.
   
   Historically, though, the reason these fields are all pointers is because originally this was going to be the first endpoint to allow PATCH requests, where every property is optional by definition. That didn't pan out because there was prerequisite work that needed to be done to support PATCH safely, but I guess I never changed the fields back from pointers.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645821712



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       Ahh, interestingly, I believe it is spaced correclty in the source code, but I deleted the comments between each of the properties for brevity, which resulted in misaligned tags. I can/will fix this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651149780



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`

Review comment:
       Removed `omitempty`. I originally thought we could aim for backwards compatibility. Also, whether it was an explicit column in the DB I figured it would default to `NULL`. However, with the newer changes this will be explicit and no longer optional.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645176789



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Oh excellent question. What are your thoughts? I suppose I would probably make this a bit more strict.
   
   To be explicitly correct, the caller would have to send either `refetch`, `refresh`, or nothing at all (which would default to `refresh`). Any other value would result in a 400. I think this is probably the best approach. The response can contain a helpful message to why it was rejected to assist any callers in the future.
   
   Should it be case *in*sensitive?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r648544088



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",
+			"level":"success"
+		}
+	],
+	"response":{
+		"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg",
+		"createdBy":"admin",
+		"deliveryService":"amc-live",
+		"id":1,
+		"keyword":"PURGE",
+		"parameters":"TTL:24h,invalidationType:refresh",
+		"startTime":"2021-06-02 09:23:21-06"
+	}
+}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 

Review comment:
       No, because really the feature flag is for `t3c` since it's generating the ATS config. I guess it could be a `t3c` option, but I'm not sure that's better than our tried and true method of global profile params. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645172135



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {

Review comment:
       Hm. How do you mean? 

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```

Review comment:
       Good call!

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Oh excellent question. What are your thoughts? I suppose I would probably make this a bit more strict.
   
   To be explicitly correct, the caller would have to send either `refetch`, `refresh`, or nothing at all (which would default to `refresh`). Any other value would result in a 400. I think this is probably the best approach. The response can contain a helpful message to why it was rejected to assist any callers in the future.
   
   Should it be case *in*sensitive?
   

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h""startTime":"2021-06-02 09:23:21-06"}}
+```
+
+Sample new response (includes the `invalidationType` on parameters field):
+```
+{"alerts":[{"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC","level":"success"}],"response":{"assetUrl":"http://amc-linear-origin.local.tld/path/.*\\.jpeg","createdBy":"admin","deliveryService":"amc-live","id":1,"keyword":"PURGE","parameters":"TTL:24h,invalidationType:refresh","startTime":"2021-06-02 09:23:21-06"}}
+```
+
+___
+
+> Note: There are still 1.x routes that reference `UserInvalidationJob`, such as 
+		`user/current/jobs(/|\.json/?)?$`
+		`user/current/jobs(/|\.json/?)?$`
+		These routes are currently deprecated and the corresponding `structs` will be removed in a future release as well.
+
+#### Client Impact
+
+Likewise with Traffic Portal, the `go` clients will need to be updated to provide this additional functionality. Since an additional field has been added to `InvalidationJobInput` in `go-tc` lib, this can be set by the client as well. If left unset, it will default to "refresh".
+
+#### Data Model / Database Impact
+
+
+The current column `parameters` will now contain a cskv (comma separated key value) string. Currently it only stores the `TTL` for the invalidation request:
+```
+TTL:48h
+```
+
+Moving forward, this column will also contain the type of cache invalidation. For instance, the string may read:
+```
+TTL:48h,invalidationType:refetch
+```
+
+If there is no `invalidationType` in the cskv, it is assumed to be **REFRESH**/**STALE** as it's default value keeping with the current implementation. Otherwise the `invalidationType` will be either `refetch` or `refresh`, defaulting to `refresh` during validation.
+
+*OPTIONAL, BUT RECOMMENDED*: As part of this effort, the _Boy Scout Rule_ will be applied ("Always leave the campground cleaner than you found it."). The `agent`, `status`, `asset_type`, `object_type`, and `object_name` columns will be removed. `agent` and `status` are currently hardcoded to the value of 1. Similarly, `asset_type` is never a anything besides "file". `object_type` and `object_name` are not used at all. This would require a DB migration with Goose as well. If this were not implemented, no migration is necessary.
+
+> The removal of these columns will impact the `UserInvalidationJob` struct. Even though the only endpoint utilizing this struct (v1.1) is deprecated, it is still in use.
+
+### ORT/T3C Impact
+
+The `regexrevalidatedotconfig.go` in _lib/tc-atscfg_ currently has a function called `filterJobs` that is responsible for parsing the _parameters_ information from the job. Right now, that is only parsing the **TTL**, however it would then also need to parse the optional, extra field of **invalidationType**. 
+
+```go
+type revalJob struct {
+	AssetURL string
+	PurgeEnd time.Time
+	Type     string // MISS or STALE (default)
+}
+```
+
+As we can see above, if the value **invalidationType** is missing or equal to **refresh**, then the `revalJob` struct's `Type` field will be set to **STALE**. Otherwise, if the **invalidationType** is **refetch** then the `Type` field will be set to **MISS**. The struct is already prepared for this information, it needs only to be parsed by during the `filterJobs` function call.
+
+
+### Traffic Monitor Impact
+
+N/A - No changes
+
+### Traffic Router Impact
+
+N/A - No changes
+
+### Traffic Stats Impact
+
+N/A - No changes
+
+### Traffic Vault Impact
+
+N/A - No changes
+
+### Documentation Impact
+
+This information will need to be added to the current **Forcing Content Invalidation** section under the **_Quick How To Guides_** section under the **_Administrator's Guide_**
+[Content Invalidation](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+Additionally, the **_Traffic Ops API_** `jobs` routes will need to be updated with the changes, API V1-V4.
+[V4 Jobs](https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/content_invalidation.html)
+
+### Testing Impact
+
+##### Unit tests
+
+For ORT/T3C there are already unit tests that will need to be updated to account for this contingency.
+
+Update:
+```
+github.com/apache/trafficcontrol/lib/go-atscfg/regexrevalidatedotconfig_test.go
+```
+
+There are no unit tests for invalidation jobs in `traffic_ops_golang`. This provides an opportunity to create unit tests to validate current and new functionality.
+
+Add:
+```
+github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go
+```
+
+##### Integration/E2E Tests:
+There are already existing integration tests for the various APIs (v1-v4) for Traffic Ops. Each will need to have this optional functionality tested as well.
+```
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v1/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v2/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v3/jobs_test.go
+github.com/apache/trafficcontrol/traffic_ops/testing/api/v4/jobs_test.go
+```
+
+### Performance Impact
+
+There will be no performance impact for Traffic Control.
+
+> Note: This functionality may create a performance impact on caches that implement a REFETCH/MISS manual override based on a regex.
+
+### Security Impact
+
+The validation in Traffic Ops of the `invalidationType` field will be such that it can only be explicitly set to **refetch**. Any other value (missing, malformed, wrong data type, etc.) will result in either a 400 level error or a default to **refresh**. No other permissions are modified.

Review comment:
       Oh excellent question. What are your thoughts? I suppose I would probably make this a bit more strict.
   
   To be explicitly correct, the caller would have to send either `refetch`, `refresh`, or nothing at all. By that I mean they would not include the `invalidationType` key at all (which would then default to `refresh`). If they did include the key and the value was an empty string, that would result in a 400. In fact, any other value would result in a 400. I think this is probably the best approach. The response can contain a helpful message to why it was rejected to assist any callers in the future.
   
   Should it be case *in*sensitive?
   

##########
File path: traffic_portal/test/integration/README.md
##########
@@ -1,3 +1,4 @@
+

Review comment:
       I believe this was addressed in a subsequent push. Please let me know if you're still seeing this change.

##########
File path: go.mod
##########
@@ -54,7 +54,6 @@ require (
 	github.com/miekg/dns v1.0.6-0.20180406150955-01d59357d468
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.1 // indirect
-	github.com/ogier/pflag v0.0.2-0.20201025181535-73e519546fc0

Review comment:
       I believe this was addressed in a subsequent push. Please let me know if you're still seeing this change.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       Fixed.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       Fixed

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}

Review comment:
       Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r645250714



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       Fixed.

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.

Review comment:
       Fixed

##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24,"invalidationType":"refresh"}

Review comment:
       Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] TaylorCFrey commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
TaylorCFrey commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r651148077



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,282 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derive and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24
+}
+```
+
+Which is mapped to a go `struct` in the `go-tc` lib.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Proposed
+
+Add an "InvalidationType" to signify a specific type of invalidation request. The InvalidationType is an optional field and will not break backwards compatibility with previous API versions. If the field is included, it _must_ be either "refetch" or "refresh".
+
+Body:
+```json
+{
+	"startTime":"2021-06-02T15:23:21.348Z",
+	"deliveryService":11,
+	"regex":"/path/.*\\.jpeg",
+	"ttl":24,
+	"invalidationType":"refresh"
+}
+```
+
+This struct now contains the `InvalidationType *string` field.
+```go
+type InvalidationJobInput struct {
+	DeliveryService *interface{} `json:"deliveryService"`
+	Regex *string `json:"regex"`
+	InvalidationType *string `json:"invalidationType,omitempty"`
+	StartTime *Time `json:"startTime"`
+	TTL *interface{} `json:"ttl"`
+	dsid *uint
+	ttl  *time.Duration
+}
+```
+
+##### Parsing the value
+
+Since the field is optional and existing functionality only signifies a **REFRESH**/**STALE** capability, if the field is omitted, empty, malformed, or in any way _not_ `refetch` then it will be treated as `refresh`.
+
+##### Response
+
+The response will be modified, then, to return this new value as well.
+
+Sample current response:
+```json
+{
+	"alerts":[
+		{
+			"text":"Invalidation request created for http://amc-linear-origin.local.tld/path/.*\\.jpeg, start:2021-06-02 15:23:21.348 +0000 UTC end 2021-06-03 15:23:21.348 +0000 UTC",

Review comment:
       I added some clarity to the alert message.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #5910: Blueprint/refetch

Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #5910:
URL: https://github.com/apache/trafficcontrol/pull/5910#discussion_r644989456



##########
File path: blueprints/refetch-invalidation.md
##########
@@ -0,0 +1,240 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Add REFETCH capability option for Content Invalidation
+
+## Problem Description
+
+Currently, within ATC, there is a concept of Invalidation Jobs. These Invalidation Jobs give a user the ability to queue an invalidation for a resource, primarily based on regular expressions. The invalidation is gathered and treated as though there was a cache **STALE**, allowing the cache to query the origin server to **REFRESH** the resource. However, should the cache policy still be incorrect or misconfigured, the resource could be updated on the origin server, but the cache will still receive a 304 - Not Modified HTTP status response.
+
+## Proposed Change
+
+ To address this potential conflict, a proposal to add **REFETCH** as an option for Invalidation Jobs. This will then be treated by caches as a **MISS** (rather than a **STALE**), thusly allowing the cache to retrieve the resource regardless of cache policies. The original **REFRESH**/**STALE** will be the default option, where **REFETCH**/**MISS** will be the addition.
+
+### Traffic Portal Impact
+
+##### Create and Update
+Traffic Portal will need to update the Invalidation Job to account for the different options. When creating an Invalidation Job both options will need to be present (Perhaps a radio button? Default will be the original **REFRESH**).
+
+Tooltips should be added to ensure an understanding of this feature at a high level.
+
+##### Read
+When displaying the information, the **Invalidation Requests** table current shows the `Parameters` field, so the display will be straight forward with no manipulation.
+
+However, we derived and calculate the expiration field based on the TTL. This code will need to be modified to account for the additional information contained in the `Parameters` field.
+
+### Traffic Ops Impact
+
+Both the API and the database schema will likely be updated, which in turn will result in changes downstream (such as T3C/ORT, clients) as well.
+
+#### REST API Impact
+
+No new endpoints will be required. However the current invalidation job will now include an optional field during `Create`. Invalidation jobs are added by submitting a POST to the jobs endpoint. 
+
+**POST** /api/4.0/jobs
+
+##### Current Request
+
+Body:
+```
+{"startTime":"2021-06-02T15:23:21.348Z","deliveryService":11,"regex":"/path/.*\\.jpeg","ttl":24}

Review comment:
       should also syntax highlight as JSON




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org