[PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]

["jnioche (via GitHub)" <gi...@apache.org>]

jnioche opened a new pull request, #2213:
URL: https://github.com/apache/stormcrawler/pull/2213

   Adds `StatusUpdaterBenchmark`, a small utility in `org.apache.stormcrawler.persistence` for measuring the throughput of a status updater bolt, and of the backend behind it, without running a topology.
   
   ### How it works
   
   - Instantiates the bolt from its class name and calls `prepare()` with the given configuration.
   - Reads the URLs from a file and passes them straight to the bolt's `store()` method instead of `execute()`, so that only the storage is measured: no cache of discovered URLs (it is disabled), no scheduling, no metadata filtering.
   - Waits for all the tuples to be acked or failed, then prints the counts, the total time and the average OPS. As in the URLFrontier client, the number of acked URLs and the OPS are also printed every minute.
   - Only warnings and errors are logged, so that the bolts' INFO messages don't drown the output.
   
   ### Input
   
   Same format as the URLFrontier client, optionally gzipped:
   
   - a plain URL per line, stored as `DISCOVERED` with the current date as next fetch date;
   - or a JSON `URLItem`: `discovered` items are stored as `DISCOVERED`, `known` ones as `FETCHED` with their `refetchable_from_date` as next fetch date (none if 0 or not set). The metadata are stored as they are.
   
   The URLs should be shuffled beforehand. Dumps are usually sorted by host, and with a backend that routes by host (e.g. OpenSearch with routing enabled), each bulk request then lands on only one or two shards.
   
   ### Usage
   
   It must be run with `storm local`, from the jar of a crawl project whose pom.xml has the dependency for the backend module (e.g. `stormcrawler-opensearch`):
   
   ```bash
   storm local target/crawler-1.0-SNAPSHOT.jar \
     org.apache.stormcrawler.persistence.StatusUpdaterBenchmark \
     org.apache.stormcrawler.opensearch.persistence.StatusUpdaterBolt \
     crawler-conf.yaml opensearch-conf.yaml urls.txt.gz
   ```
   
   The arguments are positional (bolt class, one or more configuration files, URL file) because the `storm` command intercepts options such as `-c` and `--conf`.
   
   ### Notes for reviewers
   
   - `store()` is protected but accessible because the class is in the same package as `AbstractStatusUpdaterBolt`.
   - The bolt gets a minimal `TopologyContext` built by the class, and a simple `IOutputCollector` counts the acks and fails.
   - Log levels are set with log4j's `Configurator`, which comes through the `provided` storm-client dependency and is present at runtime under `storm local`.
   - No unit tests. Tested manually with `StdOutStatusUpdater`, `MemoryStatusUpdater` (the harness alone reaches ~450k URLs/sec) and the OpenSearch status updater.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]

["jnioche (via GitHub)" <gi...@apache.org>]

jnioche commented on PR #2213:
URL: https://github.com/apache/stormcrawler/pull/2213#issuecomment-5916809566

   might be useful for #2212 
   
   obviously, this does not test how we get the URLs from the backend but given the write-heavy nature of crawling, the status updater bolts are often the bottleneck.
   
   I will share some stats later. 
   
   `docker run -d --rm --name opensearch-bench -p 127.0.0.1:9200:9200 -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -e bootstrap.memory_lock=true -e "OPENSEARCH_JAVA_OPTS=-Xms4g -Xmx4g" --ulimit memlock=-1:-1 --ulimit nofile=65536:65536 opensearchproject/opensearch:3.7.0`
   
   make sure we have a well configured opensearch-conf file then 
   
   `~/apache-storm-3.0.0/bin/storm  local target/ethicrawl-0.1-SNAPSHOT.jar org.apache.stormcrawler.persistence.StatusUpdaterBenchmark  org.apache.stormcrawler.persistence.MemoryStatusUpdater opensearch-conf.yaml crawler-conf.yaml /home/julien/url-frontier/outlinks.ndjson`
   
   It will be interesting to see if the `opensearch-java` module is faster than `opensearch` and how either of them compare against URLFrontier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add StatusUpdaterBenchmark to measure the performance of status updater backends [stormcrawler]

["jnioche (via GitHub)" <gi...@apache.org>]

jnioche merged PR #2213:
URL: https://github.com/apache/stormcrawler/pull/2213


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org