You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "waterWang (via GitHub)" <gi...@apache.org> on 2026/08/11 04:06:48 UTC
[PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
waterWang opened a new pull request, #11760:
URL: https://github.com/apache/seatunnel/pull/11760
## Description
Fix the pagination infinite-loop bug in the HTTP connector when placeholder replacement is enabled.
### Problem
When `use_placeholder_replacement=true` and `page_field` is a placeholder in `params` (e.g. `"pageIndex": "${pageIndex}"`), the page index never advances — the connector keeps requesting page 1 forever.
**Root cause:** In `HttpSourceReader.updateRequestParam()`, the `params` map is mutated in place. The first call replaces the `${pageIndex}` placeholder with the actual value (`"1"`). On every subsequent pagination iteration, the placeholder is no longer present in the map, so `replacePlaceholder` finds nothing to replace and the page index never increments, causing an infinite loop.
### Fix
Add a `rawParams` copy of the original params template (mirroring the existing `rawBody` handling). When `usePlaceholderReplacement` is enabled, rebuild the params map from `rawParams` on each pagination iteration so the `${pageField}` placeholder is always present to be replaced with the current page index.
Closes #11759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11760:
URL: https://github.com/apache/seatunnel/pull/11760#issuecomment-6338059656
A CI-status correction on unchanged head `ed1b54f1ef00061172e8241640a0c9e38be115fc`: my previous note described a queued/pending Build. The current [Build](https://github.com/apache/seatunnel/runs/118534111340) is now **failure**, so the old pending guidance is obsolete. Failed job families: `Run / unit-test`.
Concrete inspected log evidence (one representative JDK/OS per failed family; this does not claim every matrix leg has the identical cause):
- [Run / unit-test (11, ubuntu-latest)](https://github.com/waterWang/seatunnel/actions/runs/39387972908/job/118555605256): `LLMBooleanOutputTest.readsLegacyFactoryWrapperWithValidationDisabled` fails with `InvalidClassException` on `AbstractMultiCatalogTransform`.
Upstream/next-step check:
The archived LLM fixture is tracked in #12929; proposed fix #12938 remains open. Latest `dev` (`99ea16b9d6bdce452d4c4955581ccc7236a81bbd`) still has no explicit historical UID on `AbstractMultiCatalogTransform`. Syncing today will not fix that observed failure. Once the fix lands in `dev`, sync and rerun the affected failed jobs. Keep unrelated transform-serialization changes out of this PR.
My previous source-level conclusion remains unchanged: no outstanding source blocker from my side. This is a CI-status follow-up, not a fresh full code review or formal approval. The branch is ahead 3, behind 10, compare `diverged` against `dev`; this drift is recorded independently, not used as a generic sync request. A successful required Build and write-capable maintainer approval remain merge gates. Thanks for the contribution and for keeping the fix focused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11760:
URL: https://github.com/apache/seatunnel/pull/11760#issuecomment-5340992416
Hi @waterWang, thanks for joining the Slack channel!
Just a quick status update so this doesn't stall: no new commit has landed yet, so the three items from my previous review are still open:
1. **High** — the same in-place-mutation bug is still present for header-based placeholder pagination (`processPageMap` on `httpParameter.getHeaders()`), which is a documented, reachable path (`docs/en/connectors/source/Http.md`, "Example 2: Using page parameters in headers"). The `rawParams`-template pattern you applied for params would fix this the same way for headers.
2. **Medium (CI blocker)** — the fork's `Code style` (Spotless) check is currently failing on the lines you added; running `spotless:apply` locally and pushing the reformatted file should turn that green.
3. **Medium** — a regression test that calls `updateRequestParam`/`internalPollNext` across at least two pages for the params-placeholder case (mirroring `HttpSourceReaderInternalPollNextTest.testPageNumberPlaceHolderRequestBodyUpdate`) would directly protect the bug you're fixing here.
The core idea — snapshotting `rawParams` and rebuilding from that template instead of mutating in place — is exactly the right fix and correctly implemented for the params case, so this is close. Whenever you get a chance to push an update covering headers + the Spotless formatting + a regression test, I'll take another look right away. Thanks again for tracking down the root cause of #11759!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11760:
URL: https://github.com/apache/seatunnel/pull/11760#issuecomment-6285934125
Pushed `ed1b54f1ef0` on top of your branch, @waterWang (merge of the current `dev` plus one commit) to close the three open items from my review:
1. **Headers**: `HttpSourceReader` now also snapshots `rawHeaders` in the constructor and, when `use_placeholder_replacement` is on, rebuilds the headers from that template on every page, exactly like your `rawParams` fix. The params and headers paths share one private `rebuildFromTemplate` helper, so the logic is not duplicated.
2. **Spotless**: `./mvnw spotless:apply -pl seatunnel-connectors-v2/connector-http/connector-http-base` applied; `spotless:check` passes locally for the module.
3. **Regression tests**: `testPageNumberPlaceHolderParamsAdvanceAcrossPages` and `testPageNumberPlaceHolderHeadersAdvanceAcrossPages` in `HttpSourceReaderInternalPollNextTest` call `updateRequestParam` for page 1 and page 2 and assert that the substituted value advances to `2`; on `dev` both fail because the placeholder is gone after the first call.
Your fork did not create a Build run for this push; if none appears, please start one from the Actions tab ("Run workflow" on Build). I started a verification run of the same head on my fork: https://github.com/DanielLeens/seatunnel/actions?query=branch%3Apr11760-verify-ed1b54f1ef0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11760:
URL: https://github.com/apache/seatunnel/pull/11760#issuecomment-6339486959
Thanks for your patience. The upstream status has changed since my preceding note: [#12938](https://github.com/apache/seatunnel/pull/12938) merged into `dev` at 2026-10-22 10:26 UTC (merge `cc54bb433357b4cb45db9cd5d0743589c3111b79`). I checked that current dev contains the merge and that its diff restores the historical `AbstractMultiCatalogTransform` serialization UID and makes the construction-only config transient. My earlier instruction to wait for that fix is now superseded.
The archived-wrapper `LLMBooleanOutputTest.readsLegacyFactoryWrapperWithValidationDisabled` / `InvalidClassException` is the shared unit-test failure identified in the preceding review ([affected job](https://github.com/waterWang/seatunnel/actions/runs/39387972908/job/118555605256)). This head is still `ed1b54f1ef00061172e8241640a0c9e38be115fc`; the current `dev` comparison is `diverged`, ahead 3 / behind 12. Please sync with the latest `dev` and rerun the affected unit checks first, plus any stage whose log showed the same archived-wrapper error. Keep unrelated transform-serialization patches out of this PR. This is a targeted remedy for that known upstream failure, not a claim that syncing fixes every failed stage.
The required aggregate Build still needs a successful result on the final head. My previous source-level conclusion is unchanged. Required CI and the final write-capable maintainer review remain gates. This is a CI-status follow-up only, with no new source findings or fresh full-review claim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(http-source): preserve rawParams template for placeholder-based pagination (#11759) [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11760:
URL: https://github.com/apache/seatunnel/pull/11760#issuecomment-5283079402
Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
Please ignore the following message if you have joined in the Slack~
```
To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack
This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org