You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by as...@apache.org on 2026/09/14 23:54:33 UTC

(maka) branch fix/runtime-bare-429-bounded-retry created (now 4316c21339)

This is an automated email from the ASF dual-hosted git repository.

Astro-Han pushed a change to branch fix/runtime-bare-429-bounded-retry
in repository https://gitbox.apache.org/repos/asf/maka.git


      at 4316c21339 fix(runtime): retry bare rate limits with bounded local backoff

This branch includes the following new commits:

     new 4316c21339 fix(runtime): retry bare rate limits with bounded local backoff

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org


(maka) 01/01: fix(runtime): retry bare rate limits with bounded local backoff

Posted by as...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

Astro-Han pushed a commit to branch fix/runtime-bare-429-bounded-retry
in repository https://gitbox.apache.org/repos/asf/maka.git

commit 4316c213393881cec4a8620f2030937922714864
Author: AstroHan <le...@outlook.com>
AuthorDate: Tue Sep 15 07:53:23 2026 +0800

    fix(runtime): retry bare rate limits with bounded local backoff
    
    A 429 with no Retry-After failed the Turn immediately: retryMetadataFromFacts
    returned retryable:false for rate_limit unless the provider named a delay, so a
    commandcode gateway 429 ({code:'rate_limit_error', "temporarily unavailable,
    try again in a moment"}) ended the Turn with retry {decision:'declined',
    because:'policy'} even though the same response says to retry.
    
    The classifier is the single authority for retryability; the Turn loop only
    consumes it. That loop is already bounded (10 provider attempts per step, 1s→32s
    exponential backoff with jitter, SDK maxRetries:0), so a throttle with no named
    delay has a safe home: the same rule provider_capacity already uses — retryable,
    carrying retryAfterMs only when the header parses.
    
    PR #3115's original case still fails fast, and now for the right reason. That
    case was OpenCode Zen's FreeUsageLimitError (HTTP 429, error.type
    'FreeUsageLimitError'), an exhausted free allowance, not a throttle. It is now
    in PROVIDER_BILLING_PROVIDER_CODES, so it classifies as provider_billing and
    stops on the first attempt regardless of status, the same way insufficient_quota
    does. The blanket "no Retry-After, no retry" rule is gone with it.
    
    The numeric `status === 429` fallback in the retry branch is dropped: the
    classifier already maps a 429 status or code to rate_limit unless stronger
    structured evidence wins, and keeping the raw status here would have overridden
    provider_billing's fail-fast on a 429.
    
    Unchanged: aborts, Codex edge rejections (the transport already spent its
    budget), provider_billing, auth, and context_overflow stay non-retryable; 408 /
    409 / 5xx keep their existing rule.
    
    Generated-by: Claude Code
---
 .../runtime/src/__tests__/ai-sdk-backend.test.ts   | 71 ++++++++++++++++++++++
 .../runtime/src/__tests__/model-adapter.test.ts    |  2 +-
 .../provider-error-classification.test.ts          | 49 ++++++++++-----
 .../__tests__/provider-request-telemetry.test.ts   |  2 +-
 .../runtime/src/provider-error-classification.ts   | 13 ++--
 5 files changed, 113 insertions(+), 24 deletions(-)

diff --git a/packages/runtime/src/__tests__/ai-sdk-backend.test.ts b/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
index a8d04002fb..525333f0cb 100644
--- a/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
+++ b/packages/runtime/src/__tests__/ai-sdk-backend.test.ts
@@ -10395,6 +10395,77 @@ describe('AiSdkBackend RunTrace', () => {
     );
   });
 
+  test('retries a gateway rate limit that names no retry delay', async () => {
+    let calls = 0;
+    const model = new MockLanguageModelV4({
+      doStream: async () => {
+        calls += 1;
+        if (calls === 1) {
+          throw new APICallError({
+            message: 'Upstream model provider is temporarily unavailable.',
+            url: 'https://gateway.invalid/v1/chat/completions',
+            requestBodyValues: {},
+            statusCode: 429,
+            data: {
+              error: {
+                code: 'rate_limit_error',
+                message: 'Upstream model provider is temporarily unavailable.',
+              },
+            },
+          });
+        }
+        return {
+          stream: simulateReadableStream({
+            chunks: [
+              { type: 'stream-start', warnings: [] },
+              { type: 'text-start', id: 'text-1' },
+              { type: 'text-delta', id: 'text-1', delta: 'recovered' },
+              { type: 'text-end', id: 'text-1' },
+              {
+                type: 'finish',
+                finishReason: { unified: 'stop', raw: 'stop' },
+                usage: {
+                  inputTokens: { total: 1, noCache: 1, cacheRead: 0, cacheWrite: 0 },
+                  outputTokens: { total: 1, text: 1, reasoning: 0 },
+                },
+              },
+            ],
+            initialDelayInMs: null,
+            chunkDelayInMs: null,
+          }),
+        };
+      },
+    });
+    const backend = createBackend({
+      connection: connection(),
+      modelId: 'mock-model-id',
+      modelFactory: () => model,
+      tools: [],
+      providerRetrySleep: async () => {},
+    });
+
+    const events: SessionEvent[] = [];
+    for await (const event of backend.send({ turnId: 'turn-1', text: 'hi', context: [] })) {
+      events.push(event);
+    }
+
+    assert.equal(calls, 2);
+    assert.deepEqual(
+      events
+        .filter((event) => event.type === 'provider_retry')
+        .map(({ phase, reason }) => ({ phase, reason })),
+      [
+        { phase: 'scheduled', reason: 'rate_limit' },
+        { phase: 'started', reason: 'rate_limit' },
+      ],
+    );
+    assert.equal(
+      events.some((event) => event.type === 'error'),
+      false,
+    );
+    assert.equal(events.find((event) => event.type === 'complete')?.stopReason, 'end_turn');
+  });
+
   test('preserves a finished answer without regenerating when the provider connection stays open', async () => {
     const timers = manualWatchdogTimer();
     const assistants: AssistantMessage[] = [];
diff --git a/packages/runtime/src/__tests__/model-adapter.test.ts b/packages/runtime/src/__tests__/model-adapter.test.ts
index cae32b5590..a193670058 100644
--- a/packages/runtime/src/__tests__/model-adapter.test.ts
+++ b/packages/runtime/src/__tests__/model-adapter.test.ts
@@ -401,7 +401,7 @@ describe('ModelAdapter stream and error normalization', () => {
       kind: 'rate_limit',
       code: '429',
       message: '429 rate limit (code=429)',
-      retryable: false,
+      retryable: true,
     });
     // The backend consumes the typed failure without recovering the raw
     // provider error shape.
diff --git a/packages/runtime/src/__tests__/provider-error-classification.test.ts b/packages/runtime/src/__tests__/provider-error-classification.test.ts
index 40a0f44cc8..00bd4a19e2 100644
--- a/packages/runtime/src/__tests__/provider-error-classification.test.ts
+++ b/packages/runtime/src/__tests__/provider-error-classification.test.ts
@@ -55,7 +55,7 @@ describe('Provider error classification', () => {
       httpStatus: 429,
       providerCode: 'rate_limit_exceeded',
       providerRequestId: 'req-123',
-      retryable: false,
+      retryable: true,
     });
     const serialized = JSON.stringify(diagnostic);
     assert.doesNotMatch(serialized, /secret|private|authorization|request body/i);
@@ -306,35 +306,54 @@ describe('Provider error classification', () => {
     assert.partialDeepStrictEqual(providerModelFailure(failure), { retryable: true });
   });
 
-  test('does not retry a bare rate limit marked retryable by the AI SDK', () => {
-    const rateLimit = Object.assign(new Error('Rate limit exceeded'), {
+  test('retries a rate limit with or without a provider-named delay', () => {
+    // A gateway throttle with no Retry-After: the local bounded backoff owns
+    // the pacing, so the failure must not carry a delay it does not know.
+    const bareRateLimit = Object.assign(new Error('Upstream model provider is unavailable'), {
       name: 'AI_APICallError',
-      isRetryable: true,
       statusCode: 429,
+      data: {
+        error: {
+          code: 'rate_limit_error',
+          message: 'Upstream model provider is temporarily unavailable. Please try again.',
+        },
+      },
     });
-
-    assert.partialDeepStrictEqual(providerModelFailure(rateLimit), { retryable: false });
-  });
-
-  test('retries a rate limit only when the provider names a retry delay', () => {
-    const bareRateLimit = Object.assign(new Error('Rate limit exceeded'), {
+    const delayedRateLimit = Object.assign(new Error('Too many requests'), {
       name: 'AI_APICallError',
       statusCode: 429,
-      data: { error: { code: 'FreeUsageLimitError', message: 'Rate limit exceeded' } },
+      responseHeaders: { 'retry-after': '40' },
     });
-    const delayedRateLimit = Object.assign(new Error('Too many requests'), {
+    const malformedDelay = Object.assign(new Error('Too many requests'), {
       name: 'AI_APICallError',
       statusCode: 429,
-      responseHeaders: { 'retry-after': '40' },
+      responseHeaders: { 'retry-after': 'not-a-delay' },
     });
 
-    assert.partialDeepStrictEqual(providerModelFailure(bareRateLimit), { retryable: false });
+    assert.deepEqual(providerModelFailure(bareRateLimit).retryAfterMs, undefined);
+    assert.partialDeepStrictEqual(providerModelFailure(bareRateLimit), { retryable: true });
     assert.partialDeepStrictEqual(providerModelFailure(delayedRateLimit), {
       retryable: true,
       retryAfterMs: 40_000,
     });
-    assert.equal(providerFailureDiagnostic(bareRateLimit).retryable, false);
+    assert.deepEqual(providerModelFailure(malformedDelay).retryAfterMs, undefined);
+    assert.partialDeepStrictEqual(providerModelFailure(malformedDelay), { retryable: true });
+    assert.equal(providerFailureDiagnostic(bareRateLimit).retryable, true);
     assert.equal(providerFailureDiagnostic(delayedRateLimit).retryable, true);
+    assert.equal(providerFailureDiagnostic(malformedDelay).retryable, true);
+  });
+
+  test('an exhausted free tier on 429 is billing, not a throttle to retry', () => {
+    // OpenCode Zen reports the exhausted free allowance as error.type on a 429.
+    const freeUsageLimit = Object.assign(new Error('Rate limit exceeded'), {
+      name: 'AI_APICallError',
+      statusCode: 429,
+      data: { error: { type: 'FreeUsageLimitError', message: 'Rate limit exceeded' } },
+    });
+
+    assert.equal(classifyError(freeUsageLimit), 'provider_billing');
+    assert.partialDeepStrictEqual(providerModelFailure(freeUsageLimit), { retryable: false });
+    assert.equal(providerFailureDiagnostic(freeUsageLimit).retryable, false);
   });
 
   test('classifies provider capacity errors and retries with backoff', () => {
diff --git a/packages/runtime/src/__tests__/provider-request-telemetry.test.ts b/packages/runtime/src/__tests__/provider-request-telemetry.test.ts
index 8ce6effb23..72fdac6798 100644
--- a/packages/runtime/src/__tests__/provider-request-telemetry.test.ts
+++ b/packages/runtime/src/__tests__/provider-request-telemetry.test.ts
@@ -810,7 +810,7 @@ describe('canonical model-call accounting', () => {
     assert.equal(attempt.httpStatus, 429);
     assert.equal(attempt.providerCode, 'rate_limit_exceeded');
     assert.equal(attempt.providerRequestId, 'req-compact-1');
-    assert.equal(attempt.retryable, false);
+    assert.equal(attempt.retryable, true);
     assert.doesNotMatch(
       JSON.stringify(attempt),
       /private request body|private response body|private prompt/i,
diff --git a/packages/runtime/src/provider-error-classification.ts b/packages/runtime/src/provider-error-classification.ts
index 443ce12a7f..b8c86be461 100644
--- a/packages/runtime/src/provider-error-classification.ts
+++ b/packages/runtime/src/provider-error-classification.ts
@@ -71,6 +71,7 @@ const PROVIDER_BILLING_PROVIDER_CODES: ReadonlySet<string> = new Set([
   'insufficient_quota', // OpenAI & OpenAI-compatible: error.code
   'insufficient_balance', // DeepSeek: error.code
   'quota_exceeded', // OpenAI-compatible variants: error.code
+  'freeusagelimiterror', // OpenCode Zen free tier exhausted (HTTP 429): error.type
 ]);
 
 /**
@@ -227,18 +228,16 @@ function retryMetadataFromFacts(
 
   const status = Number(evidence.statusCode || evidence.code);
   const retryAfterMs = parseRetryAfterMs(facts.responseHeaders ?? {});
-  if (errorClass === 'provider_capacity') {
-    // Capacity is transient even when the provider sends a malformed delay;
-    // fall back to the adapter's bounded local backoff in that case.
+  if (errorClass === 'provider_capacity' || errorClass === 'rate_limit') {
+    // A throttle is transient whether or not the provider names a delay:
+    // gateways return bare 429s, and the adapter's bounded local backoff
+    // covers them. An exhausted account window is NOT reachable here — its
+    // structured code classifies as provider_billing above and fails fast.
     return {
       retryable: true,
       ...(retryAfterMs !== undefined && retryAfterMs !== null ? { retryAfterMs } : {}),
     };
   }
-  if (errorClass === 'rate_limit' || status === 429) {
-    if (retryAfterMs === undefined || retryAfterMs === null) return { retryable: false };
-    return { retryable: true, retryAfterMs };
-  }
   const retryable =
     errorClass === 'stream_truncated' ||
     errorClass === 'network' ||


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org