(tooling-agents) 03/14: Lower Haiku relevance concurrency for default Bedrock quota

[ak...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git

commit 05e1df1d0a3d886cb15a5c0e095e4e1d88596452
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Sun Jun 7 21:38:47 2026 +0000

    Lower Haiku relevance concurrency for default Bedrock quota
    
    Two changes targeting the Haiku throttling (retries hitting 4/5 and 5/5
    on the default account quota):
    
    - asvs_bundle had the same bug asvs_audit originally had: its Haiku
      relevance filter_batch was gated by sonnet_semaphore (5-wide) with no
      stagger, so it fired Haiku calls 5-at-once into the same instant. Give
      it a dedicated haiku_semaphore (default 2) and the same jittered startup
      stagger audit uses.
    - Lower asvs_audit HAIKU_CONCURRENCY default 3 -> 2 to match.
    
    Both tunable via the HAIKU_CONCURRENCY env var; raise after a Bedrock
    quota increase. This reduces collisions but does NOT raise the ceiling —
    on default quota a quota-increase request is still the real fix, and
    HAIKU_CONCURRENCY=1 is the most aggressive software lever.
---
 ASVS/agents/asvs_audit.py  | 11 ++++++-----
 ASVS/agents/asvs_bundle.py | 20 +++++++++++++++++++-
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/ASVS/agents/asvs_audit.py b/ASVS/agents/asvs_audit.py
index b9376ee..42c1a20 100644
--- a/ASVS/agents/asvs_audit.py
+++ b/ASVS/agents/asvs_audit.py
@@ -418,11 +418,12 @@ This means one of the following:
         SONNET_CONCURRENCY = int(os.environ.get("SONNET_CONCURRENCY", "5"))
         # Haiku relevance-scoring (Step 2) gets its own, lower concurrency.
         # It previously borrowed sonnet_semaphore (5-wide), which the
-        # Bedrock Haiku per-minute quota can't sustain across ~100 batches
-        # — every call throttled on attempt 1 and survived only via
-        # call_llm's retry/backoff. 3-wide stays under the quota so the
-        # batches stop paying the retry tax. Tune via HAIKU_CONCURRENCY.
-        HAIKU_CONCURRENCY = int(os.environ.get("HAIKU_CONCURRENCY", "3"))
+        # default Bedrock Haiku per-minute quota can't sustain across many
+        # batches — calls throttled to retry 4/5 and 5/5, surviving only via
+        # call_llm's backoff. Default 2 keeps it under the DEFAULT account
+        # quota. If you get a Bedrock quota increase for the Haiku inference
+        # profile, raise this via the HAIKU_CONCURRENCY env var.
+        HAIKU_CONCURRENCY = int(os.environ.get("HAIKU_CONCURRENCY", "2"))
 
         sonnet_semaphore = asyncio.Semaphore(SONNET_CONCURRENCY)
         opus_semaphore = asyncio.Semaphore(OPUS_CONCURRENCY)
diff --git a/ASVS/agents/asvs_bundle.py b/ASVS/agents/asvs_bundle.py
index 002abb6..20606d3 100644
--- a/ASVS/agents/asvs_bundle.py
+++ b/ASVS/agents/asvs_bundle.py
@@ -394,6 +394,7 @@ BATCH RESULTS TO CONSOLIDATE:
         import re
         import fnmatch
         import hashlib
+        import random
         from datetime import date
         audit_date = date.today().strftime("%b %d, %Y")
 
@@ -493,9 +494,17 @@ BATCH RESULTS TO CONSOLIDATE:
         # T2: configurable concurrency
         OPUS_CONCURRENCY = int(os.environ.get("OPUS_CONCURRENCY", "4"))
         SONNET_CONCURRENCY = int(os.environ.get("SONNET_CONCURRENCY", "5"))
+        # Haiku relevance-scoring gets its own, low concurrency. It used to
+        # borrow sonnet_semaphore (5-wide), which the default Bedrock Haiku
+        # per-minute quota cannot sustain across many batches — every call
+        # throttled, often to retry 4/5 or 5/5, surviving only via
+        # call_llm's backoff. Default 2 keeps it under the default quota.
+        # Raise via HAIKU_CONCURRENCY if you get a quota increase.
+        HAIKU_CONCURRENCY = int(os.environ.get("HAIKU_CONCURRENCY", "2"))
 
         sonnet_semaphore = asyncio.Semaphore(SONNET_CONCURRENCY)
         opus_semaphore = asyncio.Semaphore(OPUS_CONCURRENCY)
+        haiku_semaphore = asyncio.Semaphore(HAIKU_CONCURRENCY)
 
         # Cache key uses ALL bundled sections so re-runs with same bundle hit cache
         bundle_key = "+".join(sorted(asvs_sections))
@@ -788,7 +797,16 @@ FILES TO EVALUATE:
                 relevance_scores = {}
 
                 async def filter_batch(i, batch):
-                    async with sonnet_semaphore:
+                    # Startup stagger BEFORE acquiring the semaphore so a
+                    # sleeping coroutine doesn't hold one of the few Haiku
+                    # slots. gather() launches every batch at once; without
+                    # this the first HAIKU_CONCURRENCY calls hit Bedrock in
+                    # the same instant and throttle immediately. Spread the
+                    # opening burst across a few seconds.
+                    stagger = min(i * 0.15, 3.0) + random.uniform(0, 0.25)
+                    if stagger:
+                        await asyncio.sleep(stagger)
+                    async with haiku_semaphore:
                         entries_text = "".join(batch.values())
                         prompt = relevance_prompt_template + entries_text
                         messages = [{"role": "user", "content": prompt}]


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org