You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@stormcrawler.apache.org by "tballison (via GitHub)" <gi...@apache.org> on 2025/02/24 16:08:00 UTC

[D] Writing nextFetchDate to warc? [incubator-stormcrawler]

GitHub user tballison created a discussion: Writing nextFetchDate to warc?

I have a topology that writes to a dummy indexer, which then forwards to a warc writer and a status updater bolt.

The `nextFetchDate` is calculated in the status updater bolt. Is there an easy way to include the `nextFetchDate` in the warc?

GitHub link: https://github.com/apache/incubator-stormcrawler/discussions/1484

----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org

Re: [D] Writing nextFetchDate to warc? [incubator-stormcrawler]

Posted by "tballison (via GitHub)" <gi...@apache.org>.

GitHub user tballison closed the discussion with a comment: Writing nextFetchDate to warc?

W00t! Thank you. 

GitHub link: https://github.com/apache/incubator-stormcrawler/discussions/1484#discussioncomment-12341566

----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org

Re: [D] Writing nextFetchDate to warc? [incubator-stormcrawler]

Posted by "sebastian-nagel (via GitHub)" <gi...@apache.org>.

GitHub user sebastian-nagel added a comment to the discussion: Writing nextFetchDate to warc?

What about just calling `scheduler.schedule(status, metadata)` in a customized WARC writer? Alternatively, in a small bolt plugged in before the WARC writer and passing `nextFetchDate` forward in the metadata.

GitHub link: https://github.com/apache/incubator-stormcrawler/discussions/1484#discussioncomment-12310310

----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org

Re: [D] Writing nextFetchDate to warc? [incubator-stormcrawler]

Posted by "tballison (via GitHub)" <gi...@apache.org>.

GitHub user tballison closed a discussion: Writing nextFetchDate to warc?

I have a topology that writes to a dummy indexer, which then forwards to a warc writer and a status updater bolt.

The `nextFetchDate` is calculated in the status updater bolt. Is there an easy way to include the `nextFetchDate` in the warc?

GitHub link: https://github.com/apache/incubator-stormcrawler/discussions/1484

----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org