[USER] refetched content behavior

[Tim Allison <ta...@apache.org>]

Again, this is more for a user@ list.... Sorry.

I want to confirm I understand refetching correctly.

When the crawler goes to refetch a page, it adds the If-Modified-Since
and the If-None-Match (if an etag exists) headers. If the host
respects those, it will return a 200 and new content if something has
changed, otherwise it will return a non-200.

If the host doesn't respect those headers and returns exactly the same
bytes as were originally fetched with a 200, that content is returned
and written to a bolt.

In short, if we're writing to warcs, and we refetch a page that
returns a 200 and the contents are the same as we originally fetched,
we'll have two copies of the same content?

Thank you!

Best,

         Tim

Re: [USER] refetched content behavior

[Richard Zowalla <rz...@apache.org>]

I think, that there is no additional logic inside of the warc processing.
So in case you are going to re-fetch (the old content), you will have an additional copy.

(Disclaimer: not using warc)

> Am 07.02.2025 um 12:57 schrieb Tim Allison <ta...@apache.org>:
> 
> Again, this is more for a user@ list.... Sorry.
> 
> I want to confirm I understand refetching correctly.
> 
> When the crawler goes to refetch a page, it adds the If-Modified-Since
> and the If-None-Match (if an etag exists) headers. If the host
> respects those, it will return a 200 and new content if something has
> changed, otherwise it will return a non-200.
> 
> If the host doesn't respect those headers and returns exactly the same
> bytes as were originally fetched with a 200, that content is returned
> and written to a bolt.
> 
> In short, if we're writing to warcs, and we refetch a page that
> returns a 200 and the contents are the same as we originally fetched,
> we'll have two copies of the same content?
> 
> Thank you!
> 
> Best,
> 
>         Tim