You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "GabrielBBaldez (via GitHub)" <gi...@apache.org> on 2026/06/09 12:46:59 UTC

[PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

GabrielBBaldez opened a new pull request, #11028:
URL: https://github.com/apache/seatunnel/pull/11028

   ## Purpose of this pull request
   
   Adds a **Shopify Source connector** under Connector-V2, as requested in #11027.
   
   It follows the existing HTTP-based SaaS source connector pattern (e.g. `Klaviyo`, `Jira`): a thin `connector-http-shopify` module on top of `connector-http-base`. It authenticates with a Shopify Admin API access token sent in the `X-Shopify-Access-Token` header, and reads a Shopify Admin REST endpoint (orders, products, customers, …) into SeaTunnel rows, reusing the base HTTP source's pagination, retry/backoff, JSON-path and schema handling.
   
   ### Options
   
   - `url` (required) — the Shopify Admin API endpoint, e.g. `https://your-store.myshopify.com/admin/api/2024-01/products.json`
   - `access_token` (required) — sent as the `X-Shopify-Access-Token` header
   - plus the common HTTP source options (`method`, `schema`, `format`, `params`, `json_field`, `content_json`, retry/backoff, …)
   
   ### Scope
   
   This is a focused V1 aligned with the current SeaTunnel HTTP-connector conventions (per the issue: *"start as a focused batch source connector"*). Richer capabilities mentioned in the issue — a `resources` list, Shopify cursor-based pagination, and incremental extraction — can follow in a later iteration. Happy to align option names and scope during review.
   
   ### Added
   
   - `seatunnel-connectors-v2/connector-http/connector-http-shopify` module (`ShopifySource`, `ShopifySourceFactory`, `ShopifySourceOptions`, `ShopifySourceParameter`) + factory test
   - Registered in `connector-http/pom.xml`, `plugin-mapping.properties`, and `seatunnel-dist/pom.xml`
   - English and Chinese docs: `docs/{en,zh}/connectors/source/Shopify.md`
   
   ## Does this PR introduce _any_ user-facing change?
   
   Yes — adds a new `Shopify` source connector.
   
   ## How was this patch tested?
   
   Added a factory option-rule test (same as the other HTTP source connectors) and verified code style with `spotless`. The full multi-module build/IT runs in CI.
   
   ## Check list
   
   - [x] Code changed are covered with tests, or it does not need tests for reason
   - [x] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   - [x] If necessary, please update the documentation to describe the new feature.
   
   Closes #11027
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5488573160

   Thanks @davidzollo for the extra pass, and agreed on the "if CI passes" condition.
   
   Quick status check from my side: I just re-verified the current head (`7d9f3ad65c3e`) against the fork's Actions history, and nothing has changed since my last review — this is still the same run (`GabrielBBaldez/seatunnel` run `33429009901`), and the only failing job is still `Run / Code style` (job `99609503941`), i.e. the same Spotless formatting violation in the two new class-level Javadoc blocks I flagged (`ShopifySource.java`, `ShopifySourceFactory.java`). No new commit has landed on top of `76a93df9`/`7d9f3ad6` since then.
   
   So from a code standpoint I still don't see anything blocking on this head — but the "if CI passes" condition isn't satisfied yet. @GabrielBBaldez, could you run `mvn spotless:apply -pl seatunnel-connectors-v2/connector-http/connector-http-shopify -am` locally and push the formatted result? That should turn `Build` green and clear the last gate on this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5413836142

   Two of these were mine and are fixed in `664872f4`.
   
   **E2E.** Added `shopify_json_to_assert.conf` and its mockserver route, wired into `HttpIT` beside the other connectors. The mock matches only when `X-Shopify-Access-Token` is present, so the job also covers the one behaviour this connector adds over a plain HTTP source — if `buildWithConfig` stops injecting the header the request no longer matches and the test fails. It uses `content_field = "$.products.*"`, mirroring the documented configuration, since the Admin REST API answers with `{"products": [...]}` rather than a bare array.
   
   **Options table.** `headers` and `json_filed_missed_return_null` were missing from both `en` and `zh` while the option rule accepts them. Added, with a `headers` section noting that `access_token` already populates the token header.
   
   On the rest, what I found checking the tree rather than just this diff:
   
   **Pagination is real and worse than one dropped argument.** `HttpSource.createReader` passes eight arguments; the five-argument constructor fills the remainder with `null, false, 0L`, so `binaryMode` and `binaryChunkSize` go with `pageInfo`.
   
   But passing them through would not give working pagination here. `HttpSourceReader` resolves the next cursor with `JsonPath.read(pageCursorResponseField)` against the response **body**, and the Admin REST API returns its cursor in the `Link` **header**. Shopify also retired page-number paging for REST, so the non-cursor mode would re-request page one indefinitely rather than fail. Enabling the inherited pagination would replace a visible limitation with a silent wrong answer.
   
   So I documented the limitation instead of half-fixing it — `## Pagination` in both locales now states that `pageing` is accepted and ignored, and why. Three ways forward, and the choice is yours rather than mine:
   
   1. Teach `connector-http-base` to read a `Link`-header cursor, then wire it here. Benefits every connector against a header-cursor API; a separate PR.
   2. Keep the documented limitation for now and drop `pageing` from this connector's option rule, so a user setting it gets a validation error instead of silence.
   3. Shopify-specific pagination inside this module, diverging from the siblings.
   
   **The remaining points reproduce across the connector family, so I left them alone rather than fixing Shopify into an exception:** `getSourceClass()` is absent from the Klaviyo, Jira and Notion factories too; the `@Data` `toString` exposure is on `HttpParameter` in `connector-http-base`, not on `ShopifySourceParameter`, and reaches every connector that sets an auth header; no HTTP connector validates the URL scheme; and `connector-http-base` has no `429`/`Retry-After` handling at all. Happy to take any of them, here or separately — say which.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5516270922

   @SEZ9 — when you have a moment, would you mind taking another look? Your `CHANGES_REQUESTED` from 25 Aug is the only thing outstanding now, and it predates the commits that address it.
   
   Since that review, on head `301cce1ede52`:
   
   - **Pagination** is fixed in code rather than documented around: `pageing` now fails at startup with `HTTP-03`, covered by a test that fails against the previous behaviour with `Expected HttpConnectorException to be thrown, but nothing was thrown`.
   - `getSourceClass()` returns `ShopifySource.class`.
   - The `${SHOPIFY_ACCESS_TOKEN}` example explains the `-i` substitution it actually needs, in both locales.
   - The options table, the E2E job, and class Javadoc are in from the earlier round.
   - The two base-connector gaps you flagged — no scheme check, and the token reachable through `HttpParameter`'s `@Data` `toString()` — are filed as #12025 rather than fixed here, since they reach every header-authenticated HTTP connector.
   
   CI is green on that head, and @DanielLeens re-reviewed it from scratch this morning.
   
   No rush at all, and thank you again for the original pass — the pagination point in particular was the one that mattered, and you were right that documenting it was not enough.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4824395147

   Thanks for the ping. I rechecked the current head and the latest CI again.
   
   The head is still unchanged from my last Daniel pass, so I am not opening a new full review round on the same revision. The remaining gate is still CI, and the current red lane is `unit-test (8)` on the fork workflow. The failing test there is `org.apache.seatunnel.engine.client.SeaTunnelEngineClusterRoleTest#testWorkerIsFirstMemberThenGetJobDetailStatus`, which is outside the Shopify connector path.
   
   So on this unchanged revision I still do not see a new Shopify-side code blocker from Daniel's side. If a Shopify-specific failure shows up later, or if you push a new head, please ping me again and I will re-review right away.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4845343053

   Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
   
   - base branch: `dev`
   - head SHA: `cd0c3b61e53edbdb1408963f3b240f18dc3b3bbf`
   - compare status: `diverged`
   - ahead_by: `4`
   - behind_by: `94`
   - mergeable_state: `unknown`
   - CI / merge gate snapshot: `failing` (failing: `Build [failure]`)
   
   Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5414335840

   Amended — the changes above are now `bc5dfc33`, not the SHA I quoted.
   
   One addition while re-checking my own work against the option rule: the table sat next to a `## Pagination` section explaining `pageing`, but nothing led you there from the table itself, so anyone scanning the options would still have missed it. There is now a line under the table pointing at it, in both locales.
   
   I also cross-checked the table against the option rule programmatically rather than by eye: every option `getHttpBuilder()` accepts is either a row or, for `pageing`, that pointer. `common-options` and `enable_multi_lines` stay — the first is documented by every sibling connector, and the second is read by `HttpParameter.buildWithConfig` even though the rule does not list it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4686436487

   Thanks for the ping, and thanks for checking the CI signal carefully. I rechecked the current status from my side: the red `Build` is still the same repository-level gate, and I do not see a new Shopify-specific failure pattern behind your latest head. I also tried the minimal maintainer-side action here, but I do not have the repository admin permission needed to rerun that upstream `Build` job from this account. So at this point the next practical step is still a maintainer/admin rerun on the Apache side, rather than another code change in this PR. If the rerun comes back with a Shopify-specific failure, please ping me again and I will go back through the latest head in full.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5149754962

   Thanks for your contribution for Apache SeaTunnel. 
   
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5518363447

   Confirming what's already visible above: I re-reviewed `301cce1ede52` from scratch that same morning (not just the delta since my previous `CHANGES_REQUESTED`), including the registration chain, the runtime path, tests, and docs, and approved it — every item from both my earlier rounds and @SEZ9's Aug 25 pass is either genuinely fixed in the source (pagination now fails fast via `rejectUnsupportedPagination()` instead of silently truncating, `getSourceClass()`, the `${SHOPIFY_ACCESS_TOKEN}` example, the E2E gap) or correctly routed to #12025 for the two `connector-http-base`-wide concerns (scheme validation, `@Data` `toString()` token exposure) that aren't Shopify-specific. No blockers on my side.
   
   I also checked the merge gate again just now: `mergeStateStatus` is `BLOCKED` with `mergeable: MERGEABLE` and `reviewDecision: REVIEW_REQUIRED` — that's a review-count requirement from branch protection, not a live `CHANGES_REQUESTED` or a conflict. So the practical next step is what you asked: @SEZ9, since your Aug 25 review predates the fixes above, could you take another pass and update your review state on the current head? That should be enough to clear this one.
   
   Thanks again for the persistence through all the rounds here, @GabrielBBaldez.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4845802859

   Thanks @DanielLeens — done. Synced this branch with the latest `dev` and re-ran CI; same for my other two connector PRs that were behind: #11030 (Zendesk) and #11057 (MySQL-CDC).
   
   When you have the bandwidth, another pass on these would be appreciated — there's a small stack of connector work waiting in this area. (#11030 is already approved by @davidzollo and just needs a committer to merge.) Thanks for the steady reviews on these.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4662029693

   Thanks for the thorough review @DanielLeens! Addressed both points:
   
   - **Issue 1 (docs config contract):** updated `docs/en` and `docs/zh` to use `content_field` (the real `HttpSourceOptions` key) instead of `content_json`, and corrected the documented default for `format` to `text` to match the base HTTP connector. Kept `format = "json"` explicitly in the example.
   - **Issue 2:** added `ShopifySourceParameterTest` covering the `X-Shopify-Access-Token` and `Accept` headers, plus preservation of existing custom headers.
   
   The remaining `Build` red is the unrelated `kafka-connector-it` / `all-connectors-it-*` integration jobs on the fork, not the Shopify module.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4665873779

   Done — fixed the `format` default in the prose to `text` (en + zh) to match the runtime. The remaining CI failures look unrelated to this connector, as you flagged. Thanks for the careful re-review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4823297497

   @DanielLeens Just a ping here!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5483185777

   Thank you both — and @DanielLeens, thank you for pushing the `dev` merge yourself this morning. I rebased onto it rather than force-pushing, so `76a93df9` is intact.
   
   **Issue 1 (blocker) — fixed in code, not just documented.** You were right that documenting a data-loss path is not shipping a fix. `ShopifySource` now rejects `pageing` at construction:
   
   ```
   HTTP-03 The Shopify source does not support the 'pageing' option: it would be accepted and
   then ignored, so only the first page of results would be read while the job reported success.
   Remove 'pageing' from the source configuration.
   ```
   
   It is covered by a test, and I checked the test actually earns its place — against the previous code it fails with `Expected HttpConnectorException to be thrown, but nothing was thrown`, which is precisely the defect. A configuration without `pageing` still builds, so the normal path is unaffected.
   
   Real `Link`-header cursor support stays a follow-up against `connector-http-base`, as you suggested.
   
   **Issue 2 — done.** `getSourceClass()` returns `ShopifySource.class`.
   
   **Issue 4 — done, in both locales.** You were right that it was misleading, and the fix is a little more than a placeholder swap: `${SHOPIFY_ACCESS_TOKEN}` *is* valid SeaTunnel syntax, just not environment substitution. The docs now show the `-i SHOPIFY_ACCESS_TOKEN=…` invocation and say what happens without it, which seemed more useful than removing the variable.
   
   **Issue 7 — done.** Class-level Javadoc on the four classes.
   
   **Issues 3 and 5 — filed as #12025** against `connector-http-base`, with the scheme check and the `@Data` `toString()` exposure written up together, since both affect every header-authenticated HTTP connector. I offered to take them there once you say which direction you prefer for the scheme check — rejecting outright would break anyone testing against a local mock over plain HTTP, so that is a maintainer call.
   
   **Issue 6 — left alone deliberately.** Collapsing the two parameter objects touches how `HttpSource` builds its own `httpParameter`, which is more than this PR should carry; happy to open it separately if you would like it tracked.
   
   The docs and the `## Pagination` section were updated to match the new behaviour rather than the old wording — with the option now rejected, "accepted and ignored" would have been wrong the moment this landed.
   
   The four module tests pass on the rebased head. Thank you for the thoroughness of both passes; the pagination defect in particular would have been an unpleasant thing for a user to discover on their own.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4681218543

   Friendly nudge on the CI: the `Build` check is red, but the failing jobs are the usual shared-infra flakes on the fork runners (`unit-test` cluster-role / `jdbc-connectors-it` DB2), not Shopify coverage — the run has also been stuck in-progress for a while. The Shopify-specific tests and the code path are unchanged since your approval (head `cd0c3b61`).
   
   Since I can't re-run the upstream `Build` from my side without pushing a new commit (which would dismiss the approval), could a committer kick a re-run of the Build when convenient? Happy to do whatever's easiest on your end. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4854234920

   Thanks for the update. I rechecked the current head against the latest `dev`, and the source-level blockers from my previous Daniel review still stand on this branch.
   
   - base branch: `dev`
   - head SHA: `4b04162ea94ce0fbcf780ae26548b2f100ec65bb`
   - compare status: `diverged`
   - ahead_by: `5`
   - behind_by: `8`
   - CI / merge gate snapshot: `Build CANCELLED`
   
   The current failing CI signal does not look actionable on top of this stale head yet, so the lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I am happy to take another look on top of the updated head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "GabrielBBaldez (via GitHub)" <gi...@apache.org>.
GabrielBBaldez commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4686546929

   Thanks Daniel, really appreciate you digging in and trying the rerun. Understood — we'll wait on an admin-side Build rerun, and I'll ping you if anything Shopify-specific shows up. Thanks again for the thorough reviews! 🙏


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-4688794230

   Thanks for the follow-up. That matches what I see as well: the current head is unchanged, and I still do not see a Shopify-specific code blocker on this revision. At this point the remaining gate is the upstream Build rerun on the Apache side. If that rerun surfaces a connector-specific failure, please ping me and I will take another pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5412546691

   Thanks for the thorough follow-up, @SEZ9 - and for catching this within 25 minutes of my approval. I went back and traced Issue 1 end to end before responding, since it directly contradicts what I approved.
   
   **Issue 1 (`pageInfo`/`binaryMode` dropped in `ShopifySource.createReader()`) - confirmed, this is a real blocker and my approval was wrong to let it through.**
   
   Trace on the current head (`27a3a89b4c0c60c8ddde3a9e9a89e06b4b7c8d14`):
   - Base `HttpSource.createReader()` builds the reader with the full 8-arg constructor: `httpParameter, readerContext, deserializationSchema, jsonField, contentField, pageInfo, binaryMode, binaryChunkSize`.
   - `ShopifySource.createReader()` (`ShopifySource.java:44-52`) overrides this and calls the 5-arg constructor instead - no `pageInfo`, no `binaryMode`/`binaryChunkSize` - which delegates through the null-`pageInfo` overload (`HttpSourceReader.java:86-100`), leaving `pageInfoOptional = Optional.empty()`.
   - I then traced `HttpSourceReader.internalPollNext()` (`HttpSourceReader.java:410-435`): when `pageInfoOptional` is empty, it calls `pollAndCollectData()` exactly once per `pollNext()` invocation, and since `noMoreElementFlag` defaults to `true` (line 76) and is only ever mutated inside the page-aware branch of `collect()` (line 462), on a `BOUNDED` job the `finally` block sees `noMoreElementFlag == true` right after that first request and immediately calls `context.signalNoMoreElement()`.
   
   So this is not a hang or an infinite-poll bug - it is exactly what you described: a clean, "successful" job that silently stops after page 1. Since `getHttpBuilder()` still exposes `pageing` in `ShopifySourceFactory.optionRule()` (so a user configuring it hits no validation error, just silently-ignored config), and Shopify caps pages at 250 records, this is a genuine silent-truncation correctness bug on the real data path, not a docs nit. I'm treating my APPROVED review from earlier today as superseded - this needs `pageInfo`/`binaryMode` wired through (or a Shopify-specific reader override) before merge.
   
   **Issue 7 (missing `getSourceClass()` override) - also confirmed.** `HttpSourceFactory.getSourceClass()` returns `HttpSource.class`, and `ShopifySourceFactory` does not override it, so factory metadata for the `Shopify` identifier reports the wrong class versus what `createSource()` actually instantiates. Agreed this is Medium on its own, but worth fixing alongside Issue 1 while the file is being touched.
   
   I have not independently re-traced Issues 2-6 and 8 line by line in this pass, but they read as consistent with the doc/config-contract and secret-handling gaps I was already tracking in earlier rounds, and I do not have a reason to push back on any of them.
   
   To be clear on status: nothing has changed on this head since my (now superseded) approval, so all of your points remain open. I will do a full re-review once a new revision addressing Issue 1 (and ideally Issue 7) is pushed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Shopify] Add Shopify source connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11028:
URL: https://github.com/apache/seatunnel/pull/11028#issuecomment-5149756788

   Thanks for your contribution for Apache SeaTunnel. 
   
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org