You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "abhinav-phi (via GitHub)" <gi...@apache.org> on 2026/09/02 22:06:03 UTC

[PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

abhinav-phi opened a new pull request, #2117:
URL: https://github.com/apache/stormcrawler/pull/2117

   ## Summary
   
   URLFrontier [2.6](https://github.com/crawler-commons/url-frontier/releases/tag/2.6) added a batched endpoint for ingesting discovered URLs, `PutDiscovered(stream DiscoveredBatch) returns (stream BatchAck)`. The per-message cost of the streaming `PutURLs` endpoint is what caps the ingestion rate, and discovered URLs — the outlinks of the pages being parsed — are the bulk of what a crawl writes. This PR routes them through the batched endpoint, grouping up to `urlfrontier.batch.size` URLs (default 100) into one message, while known URLs (fetched, redirections, errors...) keep using the streaming mode, as suggested in #2062. The control-flow mechanism follows the best practices of the client shipped with URLFrontier ([`PutURLs.java`](https://github.com/crawler-commons/url-frontier/blob/master/client/src/main/java/crawlercommons/urlfrontier/client/PutURLs.java)).
   
   Fixes #2062
   
   ## What the bolt does now
   
   `StatusUpdaterBolt` keeps its overall structure: a `waitAck` cache holds the tuples per URL until the frontier acks them, and a semaphore bounds the messages in flight. What changes is the path each URL takes:
   
   **Discovered URLs → `PutDiscovered`.** A DISCOVERED URL is appended to a buffer instead of being sent on its own. The buffer leaves as one `DiscoveredBatch` message when:
   - it reaches `urlfrontier.batch.size` entries;
   - a known URL arrives — the outlinks buffered so far belong to the page whose status is now being updated, a natural end-of-page boundary for the batch (the release notes for 2.6 explicitly call the outlinks of a page a natural batch);
   - a partially filled batch has been open for more than a second (checked by a scheduled flusher every 100 ms), so that acks are not delayed when the crawl tails off;
   - the bolt is throttled because the semaphore is exhausted — pushing the batch out frees the permits sooner.
   
   Each batch carries a unique ID (`batch-N`); the frontier echoes it in the `BatchAck` with **one status per URL, in the order they were sent**, so every URL can be resolved against `waitAck` and its tuples acked (or failed on `FAIL`) exactly as the streaming path did. Permits are released in bulk per URL, unchanged.
   
   **Known URLs → `PutURLs`**, one message per URL, untouched behaviour.
   
   **Flow control without polling.** `store()` used to busy-poll `tryAcquire` with a sleep. It now waits on a monitor (`flow`) and is woken by `notifyAll()` from the places that change its conditions: permit releases (acks, evictions), transport on-ready notifications, and stream errors. The wait timeout (`urlfrontier.throttling.time.msec`, unchanged, 10 ms) is a backstop that also drives the periodic `waitAck.cleanUp()` which prevents the deadlock the old code guarded against. This mirrors the `isReady()`/on-ready pattern of the reference client instead of polling.
   
   **Fallback for older frontiers.** If the frontier does not implement `PutDiscovered` (i.e. predates 2.6), the stream fails with `UNIMPLEMENTED`. The bolt detects this, switches batching off permanently for its lifetime, and re-sends whatever was buffered or still in flight individually on the streaming endpoint — no URLs are lost, and a pre-2.6 frontier behaves exactly as before this PR. A dead batch stream (any other error) drops the pending batch IDs and opens a fresh stream on the next flush; the affected tuples are failed locally so Storm replays them.
   
   ## Configuration
   
   New key, documented in the module README:
   
   ```yaml
   # max number of discovered URLs per PutDiscovered message (default 100, 0 disables batching)
   urlfrontier.batch.size: 100
   ```
   
   The module now depends on `urlfrontier-API` **2.6** (bumped from 2.5). The schema stays wire-compatible with 2.5 apart from the semantics noted in the 2.6 release notes, and the fallback covers servers that do not implement the new RPC, so no frontier upgrade is strictly required — though the batching only pays off against a 2.6 server.
   
   ## Tests
   
   - `StatusUpdaterBoltTest.acknowledgesDiscoveredURLsSentInBatches` — 6 discovered URLs against a 2.6 frontier with batch size 2: all acked, ≥ 3 batches sent, nothing failed;
   - `StatusUpdaterBoltTest.sendsDiscoveredURLsIndividuallyWhenBatchingDisabled` — `urlfrontier.batch.size: 0` sends individually and never opens a batch;
   - `StatusUpdaterBoltTest.acksKnownURLsThroughStreamingEndpoint` — known URLs still ack through the streaming endpoint;
   - `StatusUpdaterBoltFallbackTest` — runs the bolt against a `crawlercommons/url-frontier:2.5` container: the `UNIMPLEMENTED` error is caught, batching is switched off, and discovered URLs are acked through the streaming fallback;
   - the existing `StatusUpdaterBoltTest` cases (ack with metadata, queue-stream emission, semaphore recovery after a frontier restart) pass unchanged on the new implementation.
   
   All 64 module tests pass locally against real frontier containers (testcontainers), checkstyle, forbiddenapis and the google-java-format check are clean. The `StatusUpdaterBolt` JaCoCo ratios configured for this module still pass.
   
   ## Notes for reviewers
   
   - **Dependency availability.** The url-frontier 2.6 release published its artifacts to Sonatype Central via a workflow that completed successfully on 2026-08-19, but the deployment is not yet visible on Maven Central (`urlfrontier-API` still lists 2.5 as latest; the portal deployment likely awaits the manual publish step, since `central-publishing-maven-plugin` defaults to `autoPublish=false`). This PR builds against the 2.6 API regardless — the protobuf/gRPC API is stable in the 2.6 tag — but CI will need `urlfrontier-API:2.6` to be resolvable; it may be worth pinging the crawler-commons maintainers to complete the release on the portal. If 2.6 cannot be published soon, an alternative is to keep the dependency at 2.5 and build the batch messages from the proto (not possible: `DiscoveredBatch`/`BatchAck` and the `putDiscovered` stub only exist in 2.6), so the version bump is inherent to the feature.
   - The `waitAck`-cache eviction semantics, the semaphore accounting, the queue-stream emission and the channel management (multi-address assignment, reconnect on `TRANSIENT_FAILURE`) are carried over unchanged from the previous implementation.
   - The `PutDiscovered` stream is deliberately not re-created on channel `TRANSIENT_FAILURE` (unlike the `PutURLs` stream): with `waitForReady` it survives connection blips, and abandoning it would orphan the batches already sent on it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#issuecomment-5574880702

   This one isn't a formatting failure like the others — `rat` dies during dependency resolution:
   
   ```
   Could not find artifact com.github.crawler-commons:urlfrontier-API:jar:2.6 in central
   ```
   
   `stormcrawler-urlfrontier` can't resolve `urlfrontier-API:2.6` from Maven Central, so the build never reaches the checks. Is 2.6 released yet? If it's only in a staging or snapshot repo, this PR can't go green on CI as it stands and will need to wait for the release, or pin a version that's actually on Central.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#issuecomment-5574399221

   Thank you for the review @rzo1 — all points addressed in 41ea0ba5 (branch rebased onto current main):
   
   **Inline 1 — interrupted flush stranded the batch.** Fixed exactly as suggested: the `InterruptedException` handler now removes the batch from `pendingBatches` and calls `failTupleLocally` for each of its URLs, so Storm replays them instead of waiting out the `waitAck` cache expiry. (The comment notes the batch was never handed to the transport, so failing locally loses nothing that reached the frontier.)
   
   **Inline 2 — `onBatchError` relied on cache eviction.** Fixed as suggested: the batches still tracked in `pendingBatches` are drained under `batchLock` and their tuples failed immediately. `disableBatchingAndResend` (the UNIMPLEMENTED fallback) already resent its items, so the two paths are now consistent: anything that did not reach the frontier is failed at once, anything that did and was not acked is retried by Storm.
   
   **Inline 3 — `timeSpentThrottling`.** The wait is now bracketed with `System.nanoTime()` and the metric is incremented with the time actually spent in `flow.wait` — an ack that wakes the wait early no longer bills the full `urlfrontier.throttling.time.msec`. The 30-second warn threshold uses the measured sum too.
   
   **Smaller points:**
   - `store()` now carries a comment stating that the `tryAcquire` outside the `flow` monitor was considered: a `notifyAll` racing between the failed acquire and the monitor entry only costs one backstop interval, it cannot deadlock.
   - Lock ordering documented on `sendLock`: code holding `batchLock` may acquire `sendLock`, never the reverse; the same-thread monitor re-entry from `newPutDiscoveredStream` under `batchLock` is noted as safe.
   - `urlfrontier.batch.size` added to `docs/src/main/asciidoc/configuration.adoc` (URLFrontier table).
   - Formatting: all four Java files are now formatted with the exact google-java-format version (1.35.0, AOSP) that git-code-format-maven-plugin 6.0 embeds, and `validate-code-format` passes locally with `-Dskip.format.code=false` — this includes the `newSingleThreadScheduledExecutor` lambda block and the two `prepare(...)` lines you flagged, which the earlier local formatter (a different google-java-format build) had left alone.
   
   **On the red `rat` job:** it fails only on dependency resolution of `com.github.crawler-commons:urlfrontier-API:2.6`, which is not on Maven Central yet — the url-frontier 2.6 publish workflow succeeded on 2026-08-19 but the deployment appears to be sitting unpublished on the Sonatype Central portal (`central-publishing-maven-plugin` defaults to `autoPublish=false`). I opened crawler-commons/url-frontier#217 reporting this with the evidence; a manual "Publish" of the pending deployment on central.sonatype.com should turn the job green without any change on our side. The code, tests and quality gates (checkstyle, forbiddenapis, format validation, JaCoCo ratios) all pass locally against the 2.6 API and the 2.6/2.5 frontier containers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#issuecomment-5583319062

   > @abhinav-phi CI is red here, could you have a look please?
   > 
   > The `rat` job fails on dependency resolution, not formatting: `com.github.crawler-commons:urlfrontier-API:jar:2.6` cannot be resolved. Maven Central currently has 2.5 as the newest release, so the 2.6 bump in `external/urlfrontier/pom.xml` has nothing to resolve against.
   > 
   > This one is blocked until 2.6 is published.
   > 
   > [Run 33688655282](https://github.com/apache/stormcrawler/actions/runs/33688655282)
   
   It should now be available, thanks for pointing it out


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#issuecomment-5559804150

   @abhinav-phi CI is red here, could you have a look please?
   
   The `rat` job fails on dependency resolution, not formatting: `com.github.crawler-commons:urlfrontier-API:jar:2.6` cannot be resolved. Maven Central currently has 2.5 as the newest release, so the 2.6 bump in `external/urlfrontier/pom.xml` has nothing to resolve against.
   
   This one is blocked until 2.6 is published.
   
   [Run 33688655282](https://github.com/apache/stormcrawler/actions/runs/33688655282)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#discussion_r3944168189


##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -418,7 +751,142 @@ public void store(
                     KnownURLItem.newBuilder().setInfo(info).setRefetchableFromDate(date).build());
         }
 
-        requestObserver.onNext(itemBuilder.setID(url).build());
+        final URLItem item = itemBuilder.setID(url).build();
+
+        // discovered URLs travel in batches on the PutDiscovered endpoint, known URLs keep
+        // using the streaming endpoint
+        if (status.equals(Status.DISCOVERED)) {
+            boolean shouldBatch;
+            boolean flushNow = false;
+            synchronized (batchLock) {
+                // re-read inside the lock: batching can be disabled concurrently by the
+                // fallback for frontiers without the PutDiscovered endpoint
+                shouldBatch = batching;
+                if (shouldBatch) {
+                    if (batchBuffer.isEmpty()) {
+                        oldestBufferedAt = System.currentTimeMillis();
+                    }
+                    batchBuffer.add(item);
+                    flushNow = batchBuffer.size() >= batchSize;
+                }
+            }
+            if (!shouldBatch) {
+                sendOnStreamingEndpoint(item);
+                return;
+            }
+            if (flushNow) {
+                flushBatch();
+            }
+            return;
+        }
+
+        if (batching) {
+            // the outlinks buffered so far belong to the page whose status is now updated:
+            // a natural boundary for the batch
+            flushBatch();
+        }
+
+        sendOnStreamingEndpoint(item);
+    }
+
+    /** Sends the buffered discovered URLs as one batch, if any. */
+    private void flushBatch() {
+        flushBatch(false);
+    }
+
+    /**
+     * Sends the buffered discovered URLs as one batch, if any.
+     *
+     * @param awaitTransport whether to wait briefly for the transport to become ready before
+     *     sending; only the flusher thread may do so, the Storm executor thread never stalls
+     */
+    private void flushBatch(boolean awaitTransport) {
+        final List<URLItem> items;
+        final StreamObserver<DiscoveredBatch> stream;
+        synchronized (batchLock) {
+            if (!batching || batchBuffer.isEmpty()) {
+                return;
+            }
+            if (batchRequestObserver == null) {
+                // the previous stream died: open a new one
+                batchRequestObserver = newPutDiscoveredStream();
+            }
+            stream = batchRequestObserver;
+            items = new ArrayList<>(batchBuffer);
+            batchBuffer.clear();
+            oldestBufferedAt = 0;
+        }
+
+        final String batchID = "batch-" + batchSequences.incrementAndGet();
+        final DiscoveredBatch.Builder batchBuilder = DiscoveredBatch.newBuilder().setID(batchID);
+        for (URLItem buffered : items) {
+            batchBuilder.addItems(buffered.getDiscovered().getInfo());
+        }
+        final DiscoveredBatch batch = batchBuilder.build();
+
+        // registered before the send so that a fast ack can never miss it
+        synchronized (batchLock) {
+            pendingBatches.put(batchID, items);
+        }
+
+        try {
+            if (awaitTransport) {
+                // follow the transport's lead: wait briefly for it to take the batch without
+                // buffering it, woken by the on-ready handler. The timeout is a backstop, not a
+                // poll interval.
+                final ClientCallStreamObserver<DiscoveredBatch> transport = batchTransport;
+                if (transport != null && !transport.isReady()) {
+                    synchronized (flow) {
+                        flow.wait(BATCH_FLUSH_DELAY_MS);
+                    }
+                }
+            }
+            synchronized (sendLock) {
+                stream.onNext(batch);
+            }
+            eventCounter.scope("batched").incrBy(items.size());
+            eventCounter.scope("batches").incrBy(1);
+            LOG.debug("Sent batch {} with {} discovered URL(s).", batchID, items.size());
+        } catch (InterruptedException e) {

Review Comment:
   This strands a batch.
   
   `pendingBatches.put(batchID, items)` on line 829 happens before the `awaitTransport` wait. If that wait is interrupted, which `disableBatchingAndResend` causes through `batchFlusher.shutdownNow()`, this catch only re-sets the interrupt flag. The batch is never sent, stays in `pendingBatches` forever, and its tuples sit in `waitAck` until cache expiry (default one hour) fails them. Storm will have timed them out long before.
   
   ```suggestion
           } catch (InterruptedException e) {
               Thread.currentThread().interrupt();
               synchronized (batchLock) {
                   pendingBatches.remove(batchID);
               }
               for (URLItem failed : items) {
                   failTupleLocally(failed.getID());
               }
   ```



##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -294,43 +615,55 @@ public void store(
         // First get processing permit. Otherwise, starvation possible.
         var hasPermit = false;
         var timeSpent = 0L;
+        boolean throttled = false;
         while (!hasPermit) {
-            try {
-                hasPermit = inFlightSemaphore.tryAcquire(throttleTimeMS, TimeUnit.MILLISECONDS);
-                if (!hasPermit) {
-                    LOG.trace(
-                            "{} messages in flight, time spent throttling {}",
-                            inFlightSemaphore.getQueueLength(),
-                            timeSpent);
-                    eventCounter.scope("timeSpentThrottling").incrBy(throttleTimeMS);
-                    timeSpent += throttleTimeMS;
-                    if (timeSpent >= 30000L) {
+            hasPermit = inFlightSemaphore.tryAcquire();
+            if (!hasPermit) {
+                throttled = true;
+                LOG.trace(
+                        "{} messages in flight, time spent throttling {}",
+                        inFlightSemaphore.getQueueLength(),
+                        timeSpent);
+                // wait for room on the monitor: woken as soon as an ack releases permits or the
+                // transport becomes ready again. The timeout is a backstop, not a poll interval.
+                synchronized (flow) {
+                    try {
+                        flow.wait(throttleTimeMS);
+                    } catch (InterruptedException e) {
                         LOG.warn(
-                                "Waiting more than {} ms for processing. There are {} permits available for {} waiting threads.",
-                                timeSpent,
-                                inFlightSemaphore.availablePermits(),
+                                "InterruptedException - (approx.) {} messages in flight.",
                                 inFlightSemaphore.getQueueLength());
-                    }
-                    // To prevent a deadlock, it is necessary to periodically clean up the waitAck
-                    // cache. Otherwise, in case of a frontier-side or connection-wise error, all
-                    // incoming URLs will after some time be all caught up in this loop without
-                    // touching the cache, possibly leading to no eviction and thus leading to no
-                    // release of inFlightSemaphore permits.
-                    waitAckLock.lock();
-                    try {
-                        waitAck.cleanUp();
-                    } finally {
-                        waitAckLock.unlock();
+                        Thread.currentThread().interrupt();
                     }
                 }
-            } catch (InterruptedException e) {
-                LOG.warn(
-                        "InterruptedException - (approx.) {} messages in flight.",
-                        inFlightSemaphore.getQueueLength());
-                Thread.currentThread().interrupt();
+                eventCounter.scope("timeSpentThrottling").incrBy(throttleTimeMS);

Review Comment:
   `timeSpentThrottling` is now incremented by the full `throttleTimeMS` even when `flow.wait` returned immediately because an ack woke it, which is the whole point of this change.
   
   Measure the elapsed time around the `wait` instead, otherwise the metric operators use to tune `urlfrontier.updater.max.messages` stops meaning anything.



##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -253,10 +403,168 @@ public void onNext(final crawlercommons.urlfrontier.Urlfrontier.AckMessage confi
             return;
         }
 
+        completeTuples(url, values, confirmation.getStatus());
+    }
+
+    /**
+     * Acknowledges a whole batch: one status per URL, in the order the batch was sent.
+     *
+     * @param confirmation the BatchAck received from the PutDiscovered endpoint
+     */
+    private void onNext(final BatchAck confirmation) {
+        if (closed) {
+            return;
+        }
+
+        final List<URLItem> items;
+        synchronized (batchLock) {
+            items = pendingBatches.remove(confirmation.getID());
+        }
+
+        if (items == null) {
+            LOG.debug("Could not find batch with ID `{}`.", confirmation.getID());
+            return;
+        }
+
+        final List<AckMessage.Status> statuses = confirmation.getStatusesList();
+        if (statuses.size() != items.size()) {
+            LOG.warn(
+                    "BatchAck {} carries {} status(es) for {} URL(s).",
+                    confirmation.getID(),
+                    statuses.size(),
+                    items.size());
+        }
+
+        // URLs without a status, e.g. on a protocol breach, are left to the waitAck eviction
+        int numStatuses = Math.min(statuses.size(), items.size());
+        for (int i = 0; i < numStatuses; i++) {
+            final String url = items.get(i).getID();
+            final List<Tuple> values = detachWaitAck(url);
+            if (values == null) {
+                LOG.debug("Could not find unacked tuple for id `{}`.", url);
+                continue;
+            }
+            completeTuples(url, values, statuses.get(i));
+        }
+    }
+
+    private void onBatchError(final Throwable t) {
+        if (closed) {
+            return;
+        }
+
+        // a frontier older than 2.6 does not know the PutDiscovered endpoint; instead of
+        // dropping the discovered URLs, send them individually on the streaming endpoint
+        // and keep doing so for the rest of the bolt's life
+        if (t instanceof StatusRuntimeException
+                && ((StatusRuntimeException) t).getStatus().getCode()
+                        == io.grpc.Status.Code.UNIMPLEMENTED) {
+            LOG.warn(
+                    "The frontier does not implement PutDiscovered (URLFrontier < 2.6) - sending discovered URLs individually on the streaming endpoint.");
+            disableBatchingAndResend();
+            return;
+        }
+
+        LOG.error("Error received on the batch stream: {}", t.getMessage());
+        LOG.debug("Error received on the batch stream", t);
+
+        // the stream is dead: forget the batches it carried, their tuples are failed by the
+        // waitAck cache eviction and replayed by Storm. A new stream is opened on the next flush.
+        synchronized (batchLock) {
+            pendingBatches.clear();
+            batchRequestObserver = null;
+            batchTransport = null;
+        }
+        synchronized (flow) {
+            flow.notifyAll();
+        }
+    }
+
+    /** Stops batching and pushes everything buffered or in flight through the streaming endpoint. */
+    private void disableBatchingAndResend() {
+        final List<URLItem> toResend = new ArrayList<>();
+        synchronized (batchLock) {
+            batching = false;
+            for (List<URLItem> items : pendingBatches.values()) {
+                toResend.addAll(items);
+            }
+            pendingBatches.clear();

Review Comment:
   Same expiry-window problem as the interrupt path, and inconsistent with the `RuntimeException` path in `flushBatch`, which calls `failTupleLocally` straight away.
   
   Relying on `waitAck` eviction means the tuples are only failed after `urlfrontier.cache.expireafter.sec`, long after Storm has timed them out.
   
   ```suggestion
           final List<URLItem> orphaned = new ArrayList<>();
           synchronized (batchLock) {
               for (List<URLItem> items : pendingBatches.values()) {
                   orphaned.addAll(items);
               }
               pendingBatches.clear();
               batchRequestObserver = null;
               batchTransport = null;
           }
           for (URLItem item : orphaned) {
               failTupleLocally(item.getID());
           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#issuecomment-5580272470

   CI status note: the `rat` job fails in dependency resolution — this branch needs `com.github.crawler-commons:urlfrontier-API:2.6` for the batched PutDiscovered client, but the latest release on Maven Central is still **2.5** (the release is tracked upstream in apache/url-frontier#217). So the red CI here is an upstream-release wait, not a code problem; `build` correctly skips while `rat` can't resolve dependencies.
   
   Happy to proceed either way:
   1. wait for the 2.6 release and let CI go green then (nothing to change on this branch), or
   2. if you'd rather land the change sooner, I can look at whether the batched endpoint is callable with the 2.5 API classes (the client is generated from the same proto, so it may compile against 2.5 with the new URL) — say the word and I'll try it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on code in PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#discussion_r3951961605


##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -253,10 +403,168 @@ public void onNext(final crawlercommons.urlfrontier.Urlfrontier.AckMessage confi
             return;
         }
 
+        completeTuples(url, values, confirmation.getStatus());
+    }
+
+    /**
+     * Acknowledges a whole batch: one status per URL, in the order the batch was sent.
+     *
+     * @param confirmation the BatchAck received from the PutDiscovered endpoint
+     */
+    private void onNext(final BatchAck confirmation) {
+        if (closed) {
+            return;
+        }
+
+        final List<URLItem> items;
+        synchronized (batchLock) {
+            items = pendingBatches.remove(confirmation.getID());
+        }
+
+        if (items == null) {
+            LOG.debug("Could not find batch with ID `{}`.", confirmation.getID());
+            return;
+        }
+
+        final List<AckMessage.Status> statuses = confirmation.getStatusesList();
+        if (statuses.size() != items.size()) {
+            LOG.warn(
+                    "BatchAck {} carries {} status(es) for {} URL(s).",
+                    confirmation.getID(),
+                    statuses.size(),
+                    items.size());
+        }
+
+        // URLs without a status, e.g. on a protocol breach, are left to the waitAck eviction
+        int numStatuses = Math.min(statuses.size(), items.size());
+        for (int i = 0; i < numStatuses; i++) {
+            final String url = items.get(i).getID();
+            final List<Tuple> values = detachWaitAck(url);
+            if (values == null) {
+                LOG.debug("Could not find unacked tuple for id `{}`.", url);
+                continue;
+            }
+            completeTuples(url, values, statuses.get(i));
+        }
+    }
+
+    private void onBatchError(final Throwable t) {
+        if (closed) {
+            return;
+        }
+
+        // a frontier older than 2.6 does not know the PutDiscovered endpoint; instead of
+        // dropping the discovered URLs, send them individually on the streaming endpoint
+        // and keep doing so for the rest of the bolt's life
+        if (t instanceof StatusRuntimeException
+                && ((StatusRuntimeException) t).getStatus().getCode()
+                        == io.grpc.Status.Code.UNIMPLEMENTED) {
+            LOG.warn(
+                    "The frontier does not implement PutDiscovered (URLFrontier < 2.6) - sending discovered URLs individually on the streaming endpoint.");
+            disableBatchingAndResend();
+            return;
+        }
+
+        LOG.error("Error received on the batch stream: {}", t.getMessage());
+        LOG.debug("Error received on the batch stream", t);
+
+        // the stream is dead: forget the batches it carried, their tuples are failed by the
+        // waitAck cache eviction and replayed by Storm. A new stream is opened on the next flush.
+        synchronized (batchLock) {
+            pendingBatches.clear();
+            batchRequestObserver = null;
+            batchTransport = null;
+        }
+        synchronized (flow) {
+            flow.notifyAll();
+        }
+    }
+
+    /** Stops batching and pushes everything buffered or in flight through the streaming endpoint. */
+    private void disableBatchingAndResend() {
+        final List<URLItem> toResend = new ArrayList<>();
+        synchronized (batchLock) {
+            batching = false;
+            for (List<URLItem> items : pendingBatches.values()) {
+                toResend.addAll(items);
+            }
+            pendingBatches.clear();

Review Comment:
   Fixed in 41ea0ba5 as suggested: the batches still tracked in `pendingBatches` are drained under `batchLock` and their tuples failed immediately, consistent with the RuntimeException path in `flushBatch`. Nothing depends on the `waitAck` cache expiry anymore.



##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -294,43 +615,55 @@ public void store(
         // First get processing permit. Otherwise, starvation possible.
         var hasPermit = false;
         var timeSpent = 0L;
+        boolean throttled = false;
         while (!hasPermit) {
-            try {
-                hasPermit = inFlightSemaphore.tryAcquire(throttleTimeMS, TimeUnit.MILLISECONDS);
-                if (!hasPermit) {
-                    LOG.trace(
-                            "{} messages in flight, time spent throttling {}",
-                            inFlightSemaphore.getQueueLength(),
-                            timeSpent);
-                    eventCounter.scope("timeSpentThrottling").incrBy(throttleTimeMS);
-                    timeSpent += throttleTimeMS;
-                    if (timeSpent >= 30000L) {
+            hasPermit = inFlightSemaphore.tryAcquire();
+            if (!hasPermit) {
+                throttled = true;
+                LOG.trace(
+                        "{} messages in flight, time spent throttling {}",
+                        inFlightSemaphore.getQueueLength(),
+                        timeSpent);
+                // wait for room on the monitor: woken as soon as an ack releases permits or the
+                // transport becomes ready again. The timeout is a backstop, not a poll interval.
+                synchronized (flow) {
+                    try {
+                        flow.wait(throttleTimeMS);
+                    } catch (InterruptedException e) {
                         LOG.warn(
-                                "Waiting more than {} ms for processing. There are {} permits available for {} waiting threads.",
-                                timeSpent,
-                                inFlightSemaphore.availablePermits(),
+                                "InterruptedException - (approx.) {} messages in flight.",
                                 inFlightSemaphore.getQueueLength());
-                    }
-                    // To prevent a deadlock, it is necessary to periodically clean up the waitAck
-                    // cache. Otherwise, in case of a frontier-side or connection-wise error, all
-                    // incoming URLs will after some time be all caught up in this loop without
-                    // touching the cache, possibly leading to no eviction and thus leading to no
-                    // release of inFlightSemaphore permits.
-                    waitAckLock.lock();
-                    try {
-                        waitAck.cleanUp();
-                    } finally {
-                        waitAckLock.unlock();
+                        Thread.currentThread().interrupt();
                     }
                 }
-            } catch (InterruptedException e) {
-                LOG.warn(
-                        "InterruptedException - (approx.) {} messages in flight.",
-                        inFlightSemaphore.getQueueLength());
-                Thread.currentThread().interrupt();
+                eventCounter.scope("timeSpentThrottling").incrBy(throttleTimeMS);

Review Comment:
   Fixed in 41ea0ba5: the wait is bracketed with `System.nanoTime()` and the metric now records the time actually spent in `flow.wait` - an ack waking the wait early no longer bills the full throttle interval. The 30-second warn threshold uses the measured sum as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on code in PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117#discussion_r3951960717


##########
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/StatusUpdaterBolt.java:
##########
@@ -418,7 +751,142 @@ public void store(
                     KnownURLItem.newBuilder().setInfo(info).setRefetchableFromDate(date).build());
         }
 
-        requestObserver.onNext(itemBuilder.setID(url).build());
+        final URLItem item = itemBuilder.setID(url).build();
+
+        // discovered URLs travel in batches on the PutDiscovered endpoint, known URLs keep
+        // using the streaming endpoint
+        if (status.equals(Status.DISCOVERED)) {
+            boolean shouldBatch;
+            boolean flushNow = false;
+            synchronized (batchLock) {
+                // re-read inside the lock: batching can be disabled concurrently by the
+                // fallback for frontiers without the PutDiscovered endpoint
+                shouldBatch = batching;
+                if (shouldBatch) {
+                    if (batchBuffer.isEmpty()) {
+                        oldestBufferedAt = System.currentTimeMillis();
+                    }
+                    batchBuffer.add(item);
+                    flushNow = batchBuffer.size() >= batchSize;
+                }
+            }
+            if (!shouldBatch) {
+                sendOnStreamingEndpoint(item);
+                return;
+            }
+            if (flushNow) {
+                flushBatch();
+            }
+            return;
+        }
+
+        if (batching) {
+            // the outlinks buffered so far belong to the page whose status is now updated:
+            // a natural boundary for the batch
+            flushBatch();
+        }
+
+        sendOnStreamingEndpoint(item);
+    }
+
+    /** Sends the buffered discovered URLs as one batch, if any. */
+    private void flushBatch() {
+        flushBatch(false);
+    }
+
+    /**
+     * Sends the buffered discovered URLs as one batch, if any.
+     *
+     * @param awaitTransport whether to wait briefly for the transport to become ready before
+     *     sending; only the flusher thread may do so, the Storm executor thread never stalls
+     */
+    private void flushBatch(boolean awaitTransport) {
+        final List<URLItem> items;
+        final StreamObserver<DiscoveredBatch> stream;
+        synchronized (batchLock) {
+            if (!batching || batchBuffer.isEmpty()) {
+                return;
+            }
+            if (batchRequestObserver == null) {
+                // the previous stream died: open a new one
+                batchRequestObserver = newPutDiscoveredStream();
+            }
+            stream = batchRequestObserver;
+            items = new ArrayList<>(batchBuffer);
+            batchBuffer.clear();
+            oldestBufferedAt = 0;
+        }
+
+        final String batchID = "batch-" + batchSequences.incrementAndGet();
+        final DiscoveredBatch.Builder batchBuilder = DiscoveredBatch.newBuilder().setID(batchID);
+        for (URLItem buffered : items) {
+            batchBuilder.addItems(buffered.getDiscovered().getInfo());
+        }
+        final DiscoveredBatch batch = batchBuilder.build();
+
+        // registered before the send so that a fast ack can never miss it
+        synchronized (batchLock) {
+            pendingBatches.put(batchID, items);
+        }
+
+        try {
+            if (awaitTransport) {
+                // follow the transport's lead: wait briefly for it to take the batch without
+                // buffering it, woken by the on-ready handler. The timeout is a backstop, not a
+                // poll interval.
+                final ClientCallStreamObserver<DiscoveredBatch> transport = batchTransport;
+                if (transport != null && !transport.isReady()) {
+                    synchronized (flow) {
+                        flow.wait(BATCH_FLUSH_DELAY_MS);
+                    }
+                }
+            }
+            synchronized (sendLock) {
+                stream.onNext(batch);
+            }
+            eventCounter.scope("batched").incrBy(items.size());
+            eventCounter.scope("batches").incrBy(1);
+            LOG.debug("Sent batch {} with {} discovered URL(s).", batchID, items.size());
+        } catch (InterruptedException e) {

Review Comment:
   Fixed in 41ea0ba5: the interrupt handler now removes the batch from `pendingBatches` and fails its tuples locally so Storm replays them - exactly as suggested. The batch had never been handed to the transport at that point, so nothing that reached the frontier is dropped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Use batch endpoint in URLFrontier module for DISCOVERED urls [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #2117:
URL: https://github.com/apache/stormcrawler/pull/2117


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org