You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Bartekszost (via GitHub)" <gi...@apache.org> on 2026/08/27 12:58:27 UTC

[PR] Core: Make REST scan planning poll retries configurable [iceberg]

Bartekszost opened a new pull request, #17846:
URL: https://github.com/apache/iceberg/pull/17846

   ## Summary
   
   * Add a configurable `rest-scan-planning.poll-num-retries` catalog property that controls the maximum number of poll attempts when waiting for async scan planning results in `RESTTableScan` (default remains 10)
   * This complements the existing `rest-scan-planning.poll-timeout-ms` property from #15863: the timeout bounds the total wait time, while this property bounds the number of poll retries. Together they let deployments that support **longer-running server-side scan plans** raise both limits so large plans are not cut off prematurely by a fixed retry count
   * Replace the previously hardcoded `MAX_RETRIES` constant in `RESTTableScan` with the configured value, and validate that the configured value is non-negative
   * Include the configured retry limit in the `RemotePlanTimeoutException` message so the failure is easier to diagnose
   
   ## Test plan
   
   * `asyncPlanningRespectsConfigurablePollRetries`: sets a low `poll-num-retries` against a server that never completes and verifies the poll loop stops after exactly the configured number of retries
   * `asyncPlanningSucceedsWithCustomRetries`: sets an explicit `poll-num-retries` and verifies async planning completes successfully
   * `asyncPlanningRejectsInvalidRetries`: verifies negative `poll-num-retries` values are rejected with `IllegalArgumentException`
   
   ---
   **AI Disclosure**
   - Model: Cursor Auto
   - Platform/Tool: Cursor
   - Human Oversight: partially reviewed
   - Prompt Summary: Make the REST scan planning poll retry count configurable via a catalog property, mirroring the existing configurable poll timeout, and add unit tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3989765964


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3872036692


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,140 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "10"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "custom_retries_success");
+    setParserContext(table);
+    assertThat(table.newScan().planFiles()).hasSize(1);

Review Comment:
   "10" is the default value for REST_SCAN_PLANNING_POLL_NUM_RETRIES, so this test can pass even if the property is accidentally ignored. To make the test actually validate configurability, set a non-default value and assert behavior that would differ from the default (e.g., complete within a small retry budget, or instrument fetch calls and assert the expected number of attempts).



##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,140 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");

Review Comment:
   The PR description calls out that the configured retry limit is included in the RemotePlanTimeoutException message, but this assertion only checks a generic substring. Consider also asserting the message includes the configured value (e.g., "maxRetries=0") so regressions in the diagnostic content are caught.



##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,10 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 

Review Comment:
   Consider adding a short comment/Javadoc describing the exact semantics of "poll-num-retries" (e.g., whether it counts retries in addition to the initial fetch attempt, and that 0 means "only one fetch attempt"), plus the valid range (non-negative). This reduces ambiguity for operators tuning the setting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3880036734


##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -61,7 +61,6 @@ class RESTTableScan extends DataTableScan {
   private static final Logger LOG = LoggerFactory.getLogger(RESTTableScan.class);
   private static final long MIN_SLEEP_MS = 1000; // Initial delay
   private static final long MAX_SLEEP_MS = 60 * 1000; // Max backoff delay (1 minute)
-  private static final int MAX_RETRIES = 10; // Max number of poll retries
   private static final double SCALE_FACTOR = 2.0; // Exponential scale factor

Review Comment:
   Good call — the remaining backoff knobs are now catalog properties as well (`poll-min-wait-ms`, `poll-max-wait-ms`, `poll-scale-factor`), with the same defaults as the old constants, validation, tests, and docs.



##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,10 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 

Review Comment:
   Added a short comment on the constant: this is extra poll attempts after the first fetch, must be >= 0, and 0 means a single fetch with no retries.



##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,140 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "10"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "custom_retries_success");
+    setParserContext(table);
+    assertThat(table.newScan().planFiles()).hasSize(1);

Review Comment:
   Good catch. The success path was using the default (10), so it would not fail if the property were ignored. Updated it to a non-default value (25), matching the timeout test which uses 30000 rather than the 5-minute default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3989882464


##########
docs/docs/rest-protocol.md:
##########
@@ -134,7 +134,9 @@ server either answers immediately with the completed result or returns a plan
 ID that the client polls until planning finishes. Large results are returned
 in batches of plan tasks that the client fetches separately. The client
 cancels a plan it no longer needs, and gives up after
-`rest-scan-planning.poll-timeout-ms` (default 5 minutes).
+`rest-scan-planning.poll-timeout-ms` (default 5 minutes) or
+`rest-scan-planning.poll-num-retries` (default 10) poll attempts, whichever

Review Comment:
   ah sorry I missed that refactor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3898579250


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   I would recommend doing `poll-num-retries` then and add more configs as we go ? this way we can wait for a longer amount of time while we plan / materialize which the other pr wanted to do ? ... lets also get @nastra's pov on this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3880337120


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1288,247 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");

Review Comment:
   Good catch, `asyncPlanningRespectsConfigurablePollRetries` now also asserts `maxRetries=0` in the `RemotePlanTimeoutException` message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988147783


##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -257,12 +256,22 @@ private CloseableIterable<FileScanTask> fetchPlanningResult() {
         "Invalid value for %s: %s (must be positive)",
         RESTCatalogProperties.REST_SCAN_PLANNING_POLL_TIMEOUT_MS,
         maxWaitTimeMs);
+    int maxRetries =
+        PropertyUtil.propertyAsInt(
+            catalogProperties,
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT);
+    Preconditions.checkArgument(
+        maxRetries >= 0,
+        "Invalid value for %s: %s (must be non-negative)",

Review Comment:
   can we add a test where we set this to a negative value and assert this msg?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3989661361


##########
docs/docs/rest-protocol.md:
##########
@@ -134,7 +134,9 @@ server either answers immediately with the completed result or returns a plan
 ID that the client polls until planning finishes. Large results are returned
 in batches of plan tasks that the client fetches separately. The client
 cancels a plan it no longer needs, and gives up after
-`rest-scan-planning.poll-timeout-ms` (default 5 minutes).
+`rest-scan-planning.poll-timeout-ms` (default 5 minutes) or
+`rest-scan-planning.poll-num-retries` (default 10) poll attempts, whichever

Review Comment:
   where did the docs go that you initially had?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3898579250


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   I would recommend doing `poll-num-retries` then and add more configs as we go ? this way we can wait for a longer amount of time ... lets also get @nastra's pov on this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3898649590


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   Dropped min/max wait and scale factor; this PR now only adds `poll-num-retries`. Backoff stays at the previous hardcoded defaults.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #17846:
URL: https://github.com/apache/iceberg/pull/17846


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "dillitz (via GitHub)" <gi...@apache.org>.
dillitz commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3879707242


##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -61,7 +61,6 @@ class RESTTableScan extends DataTableScan {
   private static final Logger LOG = LoggerFactory.getLogger(RESTTableScan.class);
   private static final long MIN_SLEEP_MS = 1000; // Initial delay
   private static final long MAX_SLEEP_MS = 60 * 1000; // Max backoff delay (1 minute)
-  private static final int MAX_RETRIES = 10; // Max number of poll retries
   private static final double SCALE_FACTOR = 2.0; // Exponential scale factor

Review Comment:
   Thinking further, at this point, wouldn't it make sense to make the other 3 retry parameters configurable as well? đŸ˜„



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3989773184


##########
docs/docs/rest-protocol.md:
##########
@@ -134,7 +134,9 @@ server either answers immediately with the completed result or returns a plan
 ID that the client polls until planning finishes. Large results are returned
 in batches of plan tasks that the client fetches separately. The client
 cancels a plan it no longer needs, and gives up after
-`rest-scan-planning.poll-timeout-ms` (default 5 minutes).
+`rest-scan-planning.poll-timeout-ms` (default 5 minutes) or
+`rest-scan-planning.poll-num-retries` (default 10) poll attempts, whichever

Review Comment:
   The properties table moved to `rest-catalog.md` on main. I put `poll-timeout-ms` and `poll-num-retries` back there, and kept a short mention in `rest-protocol.md`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3997175994


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,190 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });

Review Comment:
   this is exactly same as https://github.com/apache/iceberg/blob/0b933180d5dbe8ab603671761d95ec5922d8ecd7/core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java#L1158
   
   consider consolidating them ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3883187589


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   These are too many knobs whats the use case here ? how big is the table ? 
   
   can't we use task endpoint to fanout



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3892837197


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   The case is server-side plans that stay `SUBMITTED` long enough that the client hits the hardcoded retry budget before the plan finishes. #15863 made the timeout configurable, but `Tasks.retry(10)` can still stop polling first, so we added `poll-num-retries`.
   
   The task endpoint pages `plan-tasks` once results exist; it does not help while status is still `SUBMITTED`.
   
   If you want I can revert back to just adding `poll-num-retries` and keep the others default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988762043


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   Yep, already narrowed to just `poll-num-retries` after Prashant’s comment.



##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -61,7 +61,6 @@ class RESTTableScan extends DataTableScan {
   private static final Logger LOG = LoggerFactory.getLogger(RESTTableScan.class);
   private static final long MIN_SLEEP_MS = 1000; // Initial delay
   private static final long MAX_SLEEP_MS = 60 * 1000; // Max backoff delay (1 minute)
-  private static final int MAX_RETRIES = 10; // Max number of poll retries
   private static final double SCALE_FACTOR = 2.0; // Exponential scale factor

Review Comment:
   Yep, already narrowed to just `poll-num-retries` after Prashant’s comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988162474


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "custom_retries_success");
+    setParserContext(table);
+    assertThat(table.newScan().planFiles()).hasSize(1);
+  }
+
+  @Test
+  public void asyncPlanningRejectsInvalidRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    // re-initialize with an invalid retry count
+    catalogWithAdapter.catalog.initialize(
+        "test-invalid-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "-1"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "invalid_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("must be non-negative");

Review Comment:
   let's make sure to assert on the entire message by using `hasMessage(..)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988177518


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));

Review Comment:
   I guess the test would also succeed with a retry count of 1?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988212708


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,27 @@ private RESTCatalogProperties() {}
   public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
       TimeUnit.MINUTES.toMillis(5);
 
+  // Extra poll attempts after the first fetch of a submitted plan. Must be >= 0; 0 means a single
+  // fetch attempt with no retries.
+  public static final String REST_SCAN_PLANNING_POLL_NUM_RETRIES =
+      "rest-scan-planning.poll-num-retries";
+  public static final int REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT = 10;
+
+  public static final String REST_SCAN_PLANNING_POLL_MIN_WAIT_MS =
+      "rest-scan-planning.poll-min-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MIN_WAIT_MS_DEFAULT =
+      TimeUnit.SECONDS.toMillis(1);
+
+  public static final String REST_SCAN_PLANNING_POLL_MAX_WAIT_MS =
+      "rest-scan-planning.poll-max-wait-ms";
+  public static final long REST_SCAN_PLANNING_POLL_MAX_WAIT_MS_DEFAULT =
+      TimeUnit.MINUTES.toMillis(1);
+
+  // Exponential backoff multiplier between poll attempts. Must be >= 1.0.
+  public static final String REST_SCAN_PLANNING_POLL_SCALE_FACTOR =
+      "rest-scan-planning.poll-scale-factor";
+  public static final double REST_SCAN_PLANNING_POLL_SCALE_FACTOR_DEFAULT = 2.0;

Review Comment:
   yeah I agree with @singhpk234 here. We should only go with a single knob for now. Everything else adds too much complexity



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3880337120


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1288,247 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");

Review Comment:
   Good catch, asyncPlanningRespectsConfigurablePollRetries now also asserts maxRetries=0 in the RemotePlanTimeoutException message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988205980


##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -61,7 +61,6 @@ class RESTTableScan extends DataTableScan {
   private static final Logger LOG = LoggerFactory.getLogger(RESTTableScan.class);
   private static final long MIN_SLEEP_MS = 1000; // Initial delay
   private static final long MAX_SLEEP_MS = 60 * 1000; // Max backoff delay (1 minute)
-  private static final int MAX_RETRIES = 10; // Max number of poll retries
   private static final double SCALE_FACTOR = 2.0; // Exponential scale factor

Review Comment:
   I think this would be adding too many knobs, so making the retries configurable is fine, but making all the others adds too much complexity



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988752357


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "custom_retries_success");
+    setParserContext(table);
+    assertThat(table.newScan().planFiles()).hasSize(1);
+  }
+
+  @Test
+  public void asyncPlanningRejectsInvalidRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    // re-initialize with an invalid retry count
+    catalogWithAdapter.catalog.initialize(
+        "test-invalid-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "-1"));
+
+    RESTTable table = restTableFor(catalogWithAdapter.catalog, "invalid_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining("must be non-negative");

Review Comment:
   Done, `asyncPlanningRejectsInvalidRetries` now uses `hasMessage(...)` with the full validation string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3989647279


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));

Review Comment:
   I guess then let's configure the retry count to 2 here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3880292616


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1288,247 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits");

Review Comment:
   The assertion .hasMessageContaining("did not complete within configured limits") does not verify that the configured retry count appears in the RemotePlanTimeoutException message. The PR specifically advertises that the configured value is surfaced in the exception for diagnosability (format: "(timeout=%d ms, maxRetries=%d)"); with poll-num-retries=0 configured, adding .hasMessageContaining("maxRetries=0") would guard against regression in that interpolation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3997399823


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,190 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });

Review Comment:
   Done, the timeout and retry tests now share `catalogWithSubmittedPlanningResults` instead of duplicating the adapter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988147783


##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -257,12 +256,22 @@ private CloseableIterable<FileScanTask> fetchPlanningResult() {
         "Invalid value for %s: %s (must be positive)",
         RESTCatalogProperties.REST_SCAN_PLANNING_POLL_TIMEOUT_MS,
         maxWaitTimeMs);
+    int maxRetries =
+        PropertyUtil.propertyAsInt(
+            catalogProperties,
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES_DEFAULT);
+    Preconditions.checkArgument(
+        maxRetries >= 0,
+        "Invalid value for %s: %s (must be non-negative)",

Review Comment:
   can we add a test where we set this to a negative value and assert this msg?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Make REST scan planning poll retries configurable [iceberg]

Posted by "Bartekszost (via GitHub)" <gi...@apache.org>.
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3988757167


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,141 @@ public void asyncPlanningRejectsInvalidTimeout() {
         .hasMessageContaining("must be positive");
   }
 
+  @Test
+  public void asyncPlanningRespectsConfigurablePollRetries() {
+    // Create an adapter that always returns SUBMITTED (never completes)
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    AtomicInteger fetchAttempts = new AtomicInteger();
+    RESTCatalogAdapter adapter =
+        Mockito.spy(
+            new RESTCatalogAdapter(backendCatalog) {
+              @Override
+              public <T extends RESTResponse> T execute(
+                  HTTPRequest request,
+                  Class<T> responseType,
+                  Consumer<ErrorResponse> errorHandler,
+                  Consumer<Map<String, String>> responseHeaders,
+                  ParserContext parserContext) {
+                if (ResourcePaths.config().equals(request.path())) {
+                  return castResponse(
+                      responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+                }
+                T response =
+                    super.execute(
+                        request, responseType, errorHandler, responseHeaders, parserContext);
+                if (response instanceof LoadTableResponse) {
+                  return castResponse(
+                      responseType,
+                      withPlanningMode(
+                          (LoadTableResponse) response,
+                          RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+                }
+
+                // Override fetch responses to always return SUBMITTED so the poll never completes
+                if (response instanceof FetchPlanningResultResponse) {
+                  fetchAttempts.incrementAndGet();
+                  return castResponse(
+                      responseType,
+                      FetchPlanningResultResponse.builder()
+                          .withPlanStatus(PlanStatus.SUBMITTED)
+                          .build());
+                }
+
+                return response;
+              }
+            });
+
+    adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+    RESTCatalog catalog =
+        new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+    catalog.initialize(
+        "test-poll-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "0"));
+
+    RESTTable table = restTableFor(catalog, "poll_retries_test");
+    setParserContext(table);
+    RESTTableScan scan = restTableScanFor(table);
+
+    // With 0 retries and a server that never completes, planFiles should fail after one attempt
+    assertThatThrownBy(scan::planFiles)
+        .isInstanceOf(RemotePlanTimeoutException.class)
+        .hasMessageContaining("did not complete within configured limits")
+        .hasMessageContaining("maxRetries=0");
+    assertThat(fetchAttempts).hasValue(1);
+  }
+
+  @Test
+  public void asyncPlanningSucceedsWithCustomRetries() {
+    List<Endpoint> endpoints =
+        endpointsWithPlanning(
+            Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+            Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+            Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+    CatalogWithAdapter catalogWithAdapter =
+        catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+    catalogWithAdapter.catalog.initialize(
+        "test-custom-retries",
+        ImmutableMap.of(
+            CatalogProperties.FILE_IO_IMPL,
+            "org.apache.iceberg.inmemory.InMemoryFileIO",
+            RESTCatalogProperties.SCAN_PLANNING_MODE,
+            RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+            RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+            "25"));

Review Comment:
   Good catch. The success test now keeps the plan `SUBMITTED` for the first two fetches and only completes on the third, then asserts `fetchAttempts == 3`. A retry count of 1 would fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org