You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Bartekszost (via GitHub)" <gi...@apache.org> on 2026/08/28 10:58:07 UTC
Re: [PR] Core: Make REST scan planning poll retries and backoff configurable [iceberg]
Bartekszost commented on code in PR #17846:
URL: https://github.com/apache/iceberg/pull/17846#discussion_r3880036734
##########
core/src/main/java/org/apache/iceberg/rest/RESTTableScan.java:
##########
@@ -61,7 +61,6 @@ class RESTTableScan extends DataTableScan {
private static final Logger LOG = LoggerFactory.getLogger(RESTTableScan.class);
private static final long MIN_SLEEP_MS = 1000; // Initial delay
private static final long MAX_SLEEP_MS = 60 * 1000; // Max backoff delay (1 minute)
- private static final int MAX_RETRIES = 10; // Max number of poll retries
private static final double SCALE_FACTOR = 2.0; // Exponential scale factor
Review Comment:
Good call — the remaining backoff knobs are now catalog properties as well (`poll-min-wait-ms`, `poll-max-wait-ms`, `poll-scale-factor`), with the same defaults as the old constants, validation, tests, and docs.
##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -58,6 +58,10 @@ private RESTCatalogProperties() {}
public static final long REST_SCAN_PLANNING_POLL_TIMEOUT_MS_DEFAULT =
TimeUnit.MINUTES.toMillis(5);
Review Comment:
Added a short comment on the constant: this is extra poll attempts after the first fetch, must be >= 0, and 0 means a single fetch with no retries.
##########
core/src/test/java/org/apache/iceberg/rest/TestRESTScanPlanning.java:
##########
@@ -1284,6 +1285,140 @@ public void asyncPlanningRejectsInvalidTimeout() {
.hasMessageContaining("must be positive");
}
+ @Test
+ public void asyncPlanningRespectsConfigurablePollRetries() {
+ // Create an adapter that always returns SUBMITTED (never completes)
+ List<Endpoint> endpoints =
+ endpointsWithPlanning(
+ Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+ Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+ Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+ Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+ AtomicInteger fetchAttempts = new AtomicInteger();
+ RESTCatalogAdapter adapter =
+ Mockito.spy(
+ new RESTCatalogAdapter(backendCatalog) {
+ @Override
+ public <T extends RESTResponse> T execute(
+ HTTPRequest request,
+ Class<T> responseType,
+ Consumer<ErrorResponse> errorHandler,
+ Consumer<Map<String, String>> responseHeaders,
+ ParserContext parserContext) {
+ if (ResourcePaths.config().equals(request.path())) {
+ return castResponse(
+ responseType, ConfigResponse.builder().withEndpoints(endpoints).build());
+ }
+ T response =
+ super.execute(
+ request, responseType, errorHandler, responseHeaders, parserContext);
+ if (response instanceof LoadTableResponse) {
+ return castResponse(
+ responseType,
+ withPlanningMode(
+ (LoadTableResponse) response,
+ RESTCatalogProperties.ScanPlanningMode.SERVER.modeName()));
+ }
+
+ // Override fetch responses to always return SUBMITTED so the poll never completes
+ if (response instanceof FetchPlanningResultResponse) {
+ fetchAttempts.incrementAndGet();
+ return castResponse(
+ responseType,
+ FetchPlanningResultResponse.builder()
+ .withPlanStatus(PlanStatus.SUBMITTED)
+ .build());
+ }
+
+ return response;
+ }
+ });
+
+ adapter.setPlanningBehavior(TestPlanningBehavior.builder().asynchronous().build());
+
+ RESTCatalog catalog =
+ new RESTCatalog(SessionCatalog.SessionContext.createEmpty(), (config) -> adapter);
+ catalog.initialize(
+ "test-poll-retries",
+ ImmutableMap.of(
+ CatalogProperties.FILE_IO_IMPL,
+ "org.apache.iceberg.inmemory.InMemoryFileIO",
+ RESTCatalogProperties.SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+ RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+ "0"));
+
+ RESTTable table = restTableFor(catalog, "poll_retries_test");
+ setParserContext(table);
+ RESTTableScan scan = restTableScanFor(table);
+
+ // With 0 retries and a server that never completes, planFiles should fail after one attempt
+ assertThatThrownBy(scan::planFiles)
+ .isInstanceOf(RemotePlanTimeoutException.class)
+ .hasMessageContaining("did not complete within configured limits");
+ assertThat(fetchAttempts).hasValue(1);
+ }
+
+ @Test
+ public void asyncPlanningSucceedsWithCustomRetries() {
+ List<Endpoint> endpoints =
+ endpointsWithPlanning(
+ Endpoint.V1_SUBMIT_TABLE_SCAN_PLAN,
+ Endpoint.V1_FETCH_TABLE_SCAN_PLAN,
+ Endpoint.V1_CANCEL_TABLE_SCAN_PLAN,
+ Endpoint.V1_FETCH_TABLE_SCAN_PLAN_TASKS);
+
+ CatalogWithAdapter catalogWithAdapter =
+ catalogWithEndpoints(endpoints, TestPlanningBehavior.builder().asynchronous().build());
+
+ catalogWithAdapter.catalog.initialize(
+ "test-custom-retries",
+ ImmutableMap.of(
+ CatalogProperties.FILE_IO_IMPL,
+ "org.apache.iceberg.inmemory.InMemoryFileIO",
+ RESTCatalogProperties.SCAN_PLANNING_MODE,
+ RESTCatalogProperties.ScanPlanningMode.SERVER.modeName(),
+ RESTCatalogProperties.REST_SCAN_PLANNING_POLL_NUM_RETRIES,
+ "10"));
+
+ RESTTable table = restTableFor(catalogWithAdapter.catalog, "custom_retries_success");
+ setParserContext(table);
+ assertThat(table.newScan().planFiles()).hasSize(1);
Review Comment:
Good catch. The success path was using the default (10), so it would not fail if the property were ignored. Updated it to a non-default value (25), matching the timeout test which uses 30000 rather than the 5-minute default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org