You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2026/06/15 10:45:23 UTC
(hop) branch main updated: loops in REST metadata type and transform #7231 (#7233)
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new d8516c16d2 loops in REST metadata type and transform #7231 (#7233)
d8516c16d2 is described below
commit d8516c16d2379cee34c879dbdc99b9eb18134dc6
Author: Bart Maertens <ba...@know.bi>
AuthorDate: Mon Jun 15 12:45:18 2026 +0200
loops in REST metadata type and transform #7231 (#7233)
* initial working version of loops in the rest client transform. #7231
* fixes, code hardening, documentation and additional samples. fixes #7231
* cleanup. #7231
---
.../ROOT/pages/metadata-types/rest-connection.adoc | 34 +-
.../ROOT/pages/pipeline/transforms/rest.adoc | 31 +
...019-rest-client-github-releases-link-header.hpl | 500 ++++++++++++
...019-rest-client-github-releases-link-header.hwf | 96 +++
.../metadata/restconnection/github-releases.json | 10 +
.../0102-rest-client-github-pulls-link-header.hpl | 500 ++++++++++++
...n-0102-rest-client-github-pulls-link-header.hwf | 145 ++++
.../restconnection/github-hop-website.json | 10 +
.../apache/hop/metadata/rest/RestConnection.java | 174 +++-
.../hop/metadata/rest/RestConnectionEditor.java | 351 +++++++-
.../hop/metadata/rest/RestPaginationType.java | 74 ++
.../rest/messages/messages_en_US.properties | 16 +-
plugins/transforms/json/pom.xml | 1 -
plugins/transforms/rest/pom.xml | 5 +
plugins/transforms/rest/src/assembly/assembly.xml | 6 +
.../pipeline/transforms/rest/LinkHeaderPaging.java | 150 ++++
.../pipeline/transforms/rest/PagingBodyMerge.java | 155 ++++
.../apache/hop/pipeline/transforms/rest/Rest.java | 905 ++++++++++++++++++---
.../hop/pipeline/transforms/rest/RestDialog.java | 171 +++-
.../hop/pipeline/transforms/rest/RestMeta.java | 18 +
.../pipeline/transforms/rest/common/RestConst.java | 3 +
.../rest/messages/messages_en_US.properties | 17 +-
.../metadata/restconnection/github-releases.json | 10 +
.../restconnection/slack-conversations.json | 14 +
.../rest-client-github-releases-loop.hpl | 341 ++++++++
.../rest-client-slack-conversations-loop.hpl | 293 +++++++
.../transforms/rest/LinkHeaderPagingTest.java | 82 ++
.../transforms/rest/PagingBodyMergeTest.java | 63 ++
.../transforms/rest/PagingEmptySplitTest.java | 71 ++
.../rest/PagingExpressionSubstitutionTest.java | 55 ++
.../transforms/rest/PagingNextUrlTest.java | 48 ++
pom.xml | 1 +
32 files changed, 4225 insertions(+), 125 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/rest-connection.adoc b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/rest-connection.adoc
index 9e042e0696..6004458d3f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/rest-connection.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/rest-connection.adoc
@@ -73,6 +73,38 @@ Use environment variables or Hop variables (for example `{openvar}PROJECT_HOME{c
Use the `Test` button to validate the selected authentication method and SSL settings against the test or base URL. The dialog resolves variables before running the test so it matches pipeline execution.
+== Pagination
+
+The **Pagination** tab defines how the xref:pipeline/transforms/rest.adoc[REST client] transform advances through paged API responses when pagination is enabled on the transform.
+
+Pagination is configured on the connection; the REST client reads these settings and performs the HTTP loop. When the connection pagination strategy is `NONE`, the REST client performs a single request per incoming row (legacy behaviour).
+
+[options="header"]
+|===
+|Option |Description
+|Pagination strategy|How Hop obtains the next page. See the strategies below. Default is `NONE`.
+|Page / cursor query parameter name|Query parameter name for `PAGE_NUMBER` or `CURSOR` paging (for example `page` or `cursor`). Ignored by `OFFSET_LIMIT` and `LINK_HEADER`.
+|Offset query parameter name|Query parameter for the starting index in `OFFSET_LIMIT` paging. Defaults to `offset` when blank.
+|Limit query parameter name|Query parameter for the batch size in `OFFSET_LIMIT` paging. Defaults to `limit` when blank.
+|Default batch size|Page size sent with `OFFSET_LIMIT` paging. When `0`, Hop falls back to an internal default (100).
+|Cursor JsonPath|JsonPath expression against a JSON response body to read the next cursor token (`CURSOR` + JSON application type).
+|Cursor XPath|XPath expression against an XML response body to read the next cursor token (`CURSOR` + XML application type).
+|===
+
+=== Pagination strategies
+
+[options="header"]
+|===
+|Strategy |How the next page is determined
+|`NONE`|No pagination metadata is applied. Use this for single-request APIs.
+|`LINK_HEADER`|Follows the RFC 5988 `Link` response header: Hop requests the bracketed URI whose `rel` relation includes `next`. GitHub, Shopify, and similar APIs use this style. Hop stops when no `next` link is present or when the same URL would be requested again (cycle guard).
+|`OFFSET_LIMIT`|Adds offset/limit (or custom-named) query parameters. After each successful non-empty page, the offset increases by the batch size. An empty page ends the loop.
+|`PAGE_NUMBER`|Adds a page query parameter starting at `1` and increments it after each successful non-empty page. An empty page ends the loop.
+|`CURSOR`|Adds a cursor query parameter on follow-up requests. The token is extracted from the previous response with JsonPath (JSON) or XPath (XML). An empty page or missing next token ends the loop.
+|===
+
+TIP: For `LINK_HEADER`, the path on the REST client transform is still relative to `Base URL` on the first request. Subsequent requests use the absolute URL from the `next` link as returned by the API.
+
== Samples
-None
\ No newline at end of file
+link:https://github.com/apache/hop/blob/main/plugins/transforms/rest/src/main/samples/metadata/restconnection/github-releases.json[github-releases.json] — public GitHub API, `LINK_HEADER` pagination (used by link:https://github.com/apache/hop/blob/main/plugins/transforms/rest/src/main/samples/transforms/rest-client-github-releases-loop.hpl[rest-client-github-releases-loop.hpl] in the REST transform samples project).
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
index 0037969ab9..35c7a04765 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/rest.adoc
@@ -16,6 +16,8 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
+:openvar: ${
+:closevar: }
:description: The REST Client transform enables you to consume RESTful services.
= image:transforms/icons/rest.svg[REST Client transform Icon, role="image-doc-icon"] REST Client
@@ -32,6 +34,8 @@ When using a REST connection, the URL (hard-coded or accepted from a field) will
When used without a REST connection, the full URL needs to be specified.
+When pagination is enabled on this transform and the selected xref:metadata-types/rest-connection.adoc[REST connection] defines a non-`NONE` pagination strategy, Hop issues multiple HTTP requests per incoming row and can split each page into multiple output rows (see the **Pagination** tab). At **Basic** log level, each fetched page is logged (`REST pagination: fetched results page …`).
+
== Example
The REST client Transform returns a "result" field (can change the name), and the field is often used in the next transform. For example, it can be read in by a JSON input transform that extracts the fields specified on the Fields tab.
@@ -134,3 +138,30 @@ Use the Matrix Parameters tab to define matrix parameter values for POST, PUT, D
|Parameter|The field from the incoming Hop stream that contains the matrix parameter information
|Parameter|The name of the outgoing Hop field from this transform
|===
+
+=== Pagination tab
+
+Use this tab to loop over paged REST APIs. Paging behaviour (Link header, offset/limit, page number, cursor) is defined on the xref:metadata-types/rest-connection.adoc[REST connection] metadata. This tab controls whether the loop runs and how each HTTP response is turned into Hop rows.
+
+[options="header"]
+|===
+|Option|Description
+|Enable pagination loop for incoming rows|When enabled and the REST connection uses a non-`NONE` pagination strategy, Hop performs multiple HTTP requests per input row until the API has no more pages or a safeguard limit is reached. When disabled, or when the connection pagination strategy is `NONE`, Hop performs a single request (legacy behaviour).
+|Maximum page requests per input row|Hard cap on HTTP iterations per input row while following link headers, cursors, offsets, or page numbers. Leave blank or `0` to use the built-in safeguard (default 256).
+|Optional result split JsonPath/XPath|When set and **Application type** is JSON or XML, Hop emits one output row per JsonPath or XPath match instead of one row with the full response body. Typical for top-level arrays (`$[*]`) or nested collections (`$.results[*]`). Supports `{openvar}VAR{closevar}` and `%%VAR%%` substitution. This field does not apply Hop's `$[HH]` hexadecimal byte decoding, so bracketed JsonPath selectors (for example `$[*]`) are preserved.
+|===
+
+Paging stops when:
+
+* the connection strategy reports no further page (no `rel="next"` link, empty page, or missing cursor),
+* HTTP status is outside the 2xx range,
+* the maximum page request limit is reached, or
+* for `LINK_HEADER`, the next URL was already fetched (cycle guard).
+
+When a split path is configured but a page yields no matching elements, Hop fails with a clear error unless the response is a recognised empty paged collection (for example `{"results":[]}`).
+
+TIP: GitHub's public REST API requires a `User-Agent` header. Add it on the **Headers** tab (or use a Bearer token on the connection for higher rate limits). A `401` response with `No Auth` usually means an invalid `Authorization` header was sent (for example an unresolved `{openvar}GH_TOKEN{closevar}` placeholder).
+
+== Samples
+
+link:https://github.com/apache/hop/blob/main/plugins/transforms/rest/src/main/samples/transforms/rest-client-github-releases-loop.hpl[rest-client-github-releases-loop.hpl] — lists `apache/hop` GitHub releases via `LINK_HEADER` pagination (`per_page=5`), splits `$[*]`, and parses fields with a JSON Input transform. Connection metadata: link:https://github.com/apache/hop/blob/main/plugins/transforms/rest/src/main/samples/metadata/restconnection/github-releases.json[github-releases.json] (i [...]
diff --git a/integration-tests/http/0019-rest-client-github-releases-link-header.hpl b/integration-tests/http/0019-rest-client-github-releases-link-header.hpl
new file mode 100644
index 0000000000..b9327f14e5
--- /dev/null
+++ b/integration-tests/http/0019-rest-client-github-releases-link-header.hpl
@@ -0,0 +1,500 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <pipeline_version/>
+ <capture_transform_performance>N</capture_transform_performance>
+ <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+ <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ <parameter>
+ <name>PRM_MIN_RELEASE_ROWS</name>
+ <description>Minimum rows expected; with per_page=5 this proves Link-header pagination fetched more than one page.</description>
+ <default_value>6</default_value>
+ </parameter>
+ </parameters>
+ <name>0019-rest-client-github-releases-link-header</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <created_user>-</created_user>
+ <modified_user>-</modified_user>
+ <created_date>2026/05/16 14:00:00.000</created_date>
+ <modified_date>2026/05/16 14:00:00.000</modified_date>
+ </info>
+ <transform>
+ <type>Abort</type>
+ <name>Abort validation failed</name>
+ <row_threshold>0</row_threshold>
+ <message>GitHub releases pagination validation failed</message>
+ <always_log_rows>Y</always_log_rows>
+ <abort_option>ABORT_WITH_ERROR</abort_option>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1488</xloc>
+ <yloc>384</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Constant</type>
+ <name>GitHub request headers</name>
+ <fields>
+ <field>
+ <nullif>Hop-0019-rest-client-github-releases-link-header</nullif>
+ <name>user_agent</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <nullif>application/vnd.github+json</nullif>
+ <name>accept_media</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>GroupBy</type>
+ <name>aggregate release counts</name>
+ <all_rows>N</all_rows>
+ <directory>${java.io.tmpdir}</directory>
+ <prefix>grp</prefix>
+ <ignore_aggregate>N</ignore_aggregate>
+ <group/>
+ <fields>
+ <field>
+ <aggregate>row_count</aggregate>
+ <subject>tag_name</subject>
+ <type>COUNT_ANY</type>
+ <valuefield/>
+ </field>
+ <field>
+ <aggregate>distinct_tag_count</aggregate>
+ <subject>tag_name</subject>
+ <type>COUNT_DISTINCT</type>
+ <valuefield/>
+ </field>
+ </fields>
+ <add_linenr>N</add_linenr>
+ <give_back_row>N</give_back_row>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1120</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Rest</type>
+ <name>fetch apache-hop releases</name>
+ <applicationType>JSON</applicationType>
+ <connection_name>github-releases</connection_name>
+ <url>/repos/apache/hop/releases?per_page=5</url>
+ <urlInField>N</urlInField>
+ <httpPassword>Encrypted </httpPassword>
+ <preemptive>N</preemptive>
+ <method>GET</method>
+ <dynamicMethod>N</dynamicMethod>
+ <trustStorePassword>Encrypted </trustStorePassword>
+ <connectionTimeout>60000</connectionTimeout>
+ <readTimeout>60000</readTimeout>
+ <ignoreSsl>N</ignoreSsl>
+ <headers>
+ <header>
+ <field>user_agent</field>
+ <name>User-Agent</name>
+ </header>
+ <header>
+ <field>accept_media</field>
+ <name>Accept</name>
+ </header>
+ </headers>
+ <parameters/>
+ <matrixParameters/>
+ <result>
+ <name>release_json</name>
+ </result>
+ <retryTimes>0</retryTimes>
+ <retryDelayMs>200</retryDelayMs>
+ <retryStatusCodes/>
+ <retryMethods/>
+ <paginationEnabled>Y</paginationEnabled>
+ <maxPagesLoops>8</maxPagesLoops>
+ <resultSplitPath>$[*]</resultSplitPath>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>528</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>RowGenerator</type>
+ <name>generate 1 row</name>
+ <never_ending>N</never_ending>
+ <interval_in_ms>5000</interval_in_ms>
+ <row_time_field>now</row_time_field>
+ <last_time_field>FiveSecondsAgo</last_time_field>
+ <limit>1</limit>
+ <fields/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>FilterRows</type>
+ <name>pagination ok?</name>
+ <compare>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <function>=</function>
+ <conditions>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>row_count</leftvalue>
+ <function>>=</function>
+ <rightvalue>min_pages</rightvalue>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>row_count</leftvalue>
+ <function>=</function>
+ <rightvalue>distinct_tag_count</rightvalue>
+ <conditions/>
+ </condition>
+ </conditions>
+ </condition>
+ </compare>
+ <send_true_to>success</send_true_to>
+ <send_false_to>Abort validation failed</send_false_to>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1488</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>JsonInput</type>
+ <name>parse release JSON</name>
+ <include>N</include>
+ <include_field/>
+ <rownum>N</rownum>
+ <rownum_field/>
+ <limit>0</limit>
+ <valueField>release_json</valueField>
+ <IsInFields>Y</IsInFields>
+ <IsAFile>N</IsAFile>
+ <addresultfile>N</addresultfile>
+ <IsIgnoreEmptyFile>Y</IsIgnoreEmptyFile>
+ <doNotFailIfNoFile>N</doNotFailIfNoFile>
+ <ignoreMissingPath>Y</ignoreMissingPath>
+ <readurl>N</readurl>
+ <removeSourceField>Y</removeSourceField>
+ <defaultPathLeafToNull>Y</defaultPathLeafToNull>
+ <files>
+ <file>
+ <name/>
+ <filemask/>
+ <exclude_filemask/>
+ <file_required>N</file_required>
+ <include_subfolders>N</include_subfolders>
+ </file>
+ </files>
+ <fields>
+ <field>
+ <name>tag_name</name>
+ <length>-1</length>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <repeat>N</repeat>
+ <path>$.tag_name</path>
+ </field>
+ <field>
+ <name>html_url</name>
+ <length>-1</length>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <repeat>N</repeat>
+ <path>$.html_url</path>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>736</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>FilterRows</type>
+ <name>release fields ok?</name>
+ <compare>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>tag_name</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>tag_name</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>html_url</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>html_url</leftvalue>
+ <function>STARTS WITH</function>
+ <value>
+ <name>constant</name>
+ <type>String</type>
+ <text>https://github.com/apache/hop/releases/tag/</text>
+ <length>-1</length>
+ <precision>-1</precision>
+ <isnull>N</isnull>
+ <mask/>
+ </value>
+ <conditions/>
+ </condition>
+ </conditions>
+ </condition>
+ </compare>
+ <send_true_to>aggregate release counts</send_true_to>
+ <send_false_to>Abort validation failed</send_false_to>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>928</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Dummy</type>
+ <name>success</name>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1696</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>GetVariable</type>
+ <name>Get variables</name>
+ <fields>
+ <field>
+ <name>min_pages</name>
+ <variable>${PRM_MIN_RELEASE_ROWS}</variable>
+ <type>Integer</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <trim_type>none</trim_type>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1312</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <order>
+ <hop>
+ <from>generate 1 row</from>
+ <to>GitHub request headers</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>GitHub request headers</from>
+ <to>fetch apache-hop releases</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>fetch apache-hop releases</from>
+ <to>parse release JSON</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>parse release JSON</from>
+ <to>release fields ok?</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>release fields ok?</from>
+ <to>aggregate release counts</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>release fields ok?</from>
+ <to>Abort validation failed</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pagination ok?</from>
+ <to>success</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pagination ok?</from>
+ <to>Abort validation failed</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>aggregate release counts</from>
+ <to>Get variables</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Get variables</from>
+ <to>pagination ok?</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <notepads>
+ <notepad>
+ <note>Integration test: live GET /repos/apache/hop/releases with LINK_HEADER pagination (per_page=5). No golden data — assertions only.
+
+Passes when row_count >= ${PRM_MIN_RELEASE_ROWS} (default 6, so more than one page), row_count equals distinct tag_name, and tag_name/html_url look valid. Stays valid as new releases are published.
+</note>
+ <fontname>SansSerif</fontname>
+ <fontsize>10</fontsize>
+ <fontbold>N</fontbold>
+ <fontitalic>N</fontitalic>
+ <fontcolorred>14</fontcolorred>
+ <fontcolorgreen>58</fontcolorgreen>
+ <fontcolorblue>90</fontcolorblue>
+ <backgroundcolorred>201</backgroundcolorred>
+ <backgroundcolorgreen>232</backgroundcolorgreen>
+ <backgroundcolorblue>251</backgroundcolorblue>
+ <bordercolorred>14</bordercolorred>
+ <bordercolorgreen>58</bordercolorgreen>
+ <bordercolorblue>90</bordercolorblue>
+ <xloc>64</xloc>
+ <yloc>32</yloc>
+ <width>720</width>
+ <height>200</height>
+ </notepad>
+ </notepads>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git a/integration-tests/http/main-0019-rest-client-github-releases-link-header.hwf b/integration-tests/http/main-0019-rest-client-github-releases-link-header.hwf
new file mode 100644
index 0000000000..34cf20f42f
--- /dev/null
+++ b/integration-tests/http/main-0019-rest-client-github-releases-link-header.hwf
@@ -0,0 +1,96 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0019-rest-client-github-releases-link-header</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <created_user>-</created_user>
+ <modified_user>-</modified_user>
+ <created_date>2026/05/16 14:00:00.000</created_date>
+ <modified_date>2026/05/16 14:00:00.000</modified_date>
+ <workflow_version/>
+ <parameters/>
+ <actions>
+ <action>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <intervalSeconds>0</intervalSeconds>
+ <intervalMinutes>60</intervalMinutes>
+ <DayOfMonth>1</DayOfMonth>
+ <weekDay>1</weekDay>
+ <minutes>0</minutes>
+ <hour>12</hour>
+ <doNotWaitOnFirstExecution>N</doNotWaitOnFirstExecution>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <xloc>50</xloc>
+ <yloc>50</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <filename>${PROJECT_HOME}/0019-rest-client-github-releases-link-header.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <create_parent_folder>N</create_parent_folder>
+ <set_logfile>N</set_logfile>
+ <set_append_logfile>N</set_append_logfile>
+ <logfile/>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <wait_until_finished>Y</wait_until_finished>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ <parameter>
+ <name>PRM_MIN_RELEASE_ROWS</name>
+ <value>6</value>
+ <stream_name/>
+ </parameter>
+ </parameters>
+ <run_configuration>local</run_configuration>
+ <name>0019-rest-client-github-releases-link-header.hpl</name>
+ <description/>
+ <type>PIPELINE</type>
+ <attributes/>
+ <xloc>352</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>0019-rest-client-github-releases-link-header.hpl</to>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ </hops>
+ <notepads/>
+ <attributes/>
+</workflow>
diff --git a/integration-tests/http/metadata/restconnection/github-releases.json b/integration-tests/http/metadata/restconnection/github-releases.json
new file mode 100644
index 0000000000..037559e1d1
--- /dev/null
+++ b/integration-tests/http/metadata/restconnection/github-releases.json
@@ -0,0 +1,10 @@
+{
+ "auth_type": "No Auth",
+ "base_url": "https://api.github.com",
+ "test_url": "https://api.github.com/repos/apache/hop/releases?per_page=1",
+ "ignoreSsl": false,
+ "pagination_type": "LINK_HEADER",
+ "trustStorePassword": "Encrypted ",
+ "trustStoreFile": "",
+ "name": "github-releases"
+}
diff --git a/integration-tests/transforms/0102-rest-client-github-pulls-link-header.hpl b/integration-tests/transforms/0102-rest-client-github-pulls-link-header.hpl
new file mode 100644
index 0000000000..1b53de43d4
--- /dev/null
+++ b/integration-tests/transforms/0102-rest-client-github-pulls-link-header.hpl
@@ -0,0 +1,500 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <pipeline_version/>
+ <capture_transform_performance>N</capture_transform_performance>
+ <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+ <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ <parameter>
+ <name>PRM_MIN_PULL_ROWS</name>
+ <description>Minimum pull-request rows; with per_page=100 this proves Link-header pagination across multiple pages.</description>
+ <default_value>250</default_value>
+ </parameter>
+ </parameters>
+ <name>0102-rest-client-github-pulls-link-header</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <created_user>-</created_user>
+ <modified_user>-</modified_user>
+ <created_date>2026/06/06 12:00:00.000</created_date>
+ <modified_date>2026/06/06 12:00:00.000</modified_date>
+ </info>
+ <transform>
+ <type>Abort</type>
+ <name>Abort validation failed</name>
+ <row_threshold>0</row_threshold>
+ <message>GitHub hop-website pull-request pagination validation failed</message>
+ <always_log_rows>Y</always_log_rows>
+ <abort_option>ABORT_WITH_ERROR</abort_option>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1488</xloc>
+ <yloc>384</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Constant</type>
+ <name>GitHub request headers</name>
+ <fields>
+ <field>
+ <nullif>Hop-0102-rest-client-github-pulls-link-header</nullif>
+ <name>user_agent</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <nullif>application/vnd.github+json</nullif>
+ <name>accept_media</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>GroupBy</type>
+ <name>aggregate pull counts</name>
+ <all_rows>N</all_rows>
+ <directory>${java.io.tmpdir}</directory>
+ <prefix>grp</prefix>
+ <ignore_aggregate>N</ignore_aggregate>
+ <group/>
+ <fields>
+ <field>
+ <aggregate>row_count</aggregate>
+ <subject>pr_number</subject>
+ <type>COUNT_ANY</type>
+ <valuefield/>
+ </field>
+ <field>
+ <aggregate>distinct_pr_count</aggregate>
+ <subject>pr_number</subject>
+ <type>COUNT_DISTINCT</type>
+ <valuefield/>
+ </field>
+ </fields>
+ <add_linenr>N</add_linenr>
+ <give_back_row>N</give_back_row>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1120</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Rest</type>
+ <name>fetch hop-website pull requests</name>
+ <applicationType>JSON</applicationType>
+ <connection_name>github-hop-website</connection_name>
+ <url>/repos/apache/hop-website/pulls?state=all&per_page=100</url>
+ <urlInField>N</urlInField>
+ <httpPassword>Encrypted </httpPassword>
+ <preemptive>N</preemptive>
+ <method>GET</method>
+ <dynamicMethod>N</dynamicMethod>
+ <trustStorePassword>Encrypted </trustStorePassword>
+ <connectionTimeout>120000</connectionTimeout>
+ <readTimeout>120000</readTimeout>
+ <ignoreSsl>N</ignoreSsl>
+ <headers>
+ <header>
+ <field>user_agent</field>
+ <name>User-Agent</name>
+ </header>
+ <header>
+ <field>accept_media</field>
+ <name>Accept</name>
+ </header>
+ </headers>
+ <parameters/>
+ <matrixParameters/>
+ <result>
+ <name>pull_json</name>
+ </result>
+ <retryTimes>0</retryTimes>
+ <retryDelayMs>200</retryDelayMs>
+ <retryStatusCodes/>
+ <retryMethods/>
+ <paginationEnabled>Y</paginationEnabled>
+ <maxPagesLoops>5</maxPagesLoops>
+ <resultSplitPath>$[*]</resultSplitPath>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>528</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>RowGenerator</type>
+ <name>generate 1 row</name>
+ <never_ending>N</never_ending>
+ <interval_in_ms>5000</interval_in_ms>
+ <row_time_field>now</row_time_field>
+ <last_time_field>FiveSecondsAgo</last_time_field>
+ <limit>1</limit>
+ <fields/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>FilterRows</type>
+ <name>pagination ok?</name>
+ <compare>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <function>=</function>
+ <conditions>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>row_count</leftvalue>
+ <function>>=</function>
+ <rightvalue>min_pull_rows</rightvalue>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>row_count</leftvalue>
+ <function>=</function>
+ <rightvalue>distinct_pr_count</rightvalue>
+ <conditions/>
+ </condition>
+ </conditions>
+ </condition>
+ </compare>
+ <send_true_to>success</send_true_to>
+ <send_false_to>Abort validation failed</send_false_to>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1488</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>JsonInput</type>
+ <name>parse pull JSON</name>
+ <include>N</include>
+ <include_field/>
+ <rownum>N</rownum>
+ <rownum_field/>
+ <limit>0</limit>
+ <valueField>pull_json</valueField>
+ <IsInFields>Y</IsInFields>
+ <IsAFile>N</IsAFile>
+ <addresultfile>N</addresultfile>
+ <IsIgnoreEmptyFile>Y</IsIgnoreEmptyFile>
+ <doNotFailIfNoFile>N</doNotFailIfNoFile>
+ <ignoreMissingPath>Y</ignoreMissingPath>
+ <readurl>N</readurl>
+ <removeSourceField>Y</removeSourceField>
+ <defaultPathLeafToNull>Y</defaultPathLeafToNull>
+ <files>
+ <file>
+ <name/>
+ <filemask/>
+ <exclude_filemask/>
+ <file_required>N</file_required>
+ <include_subfolders>N</include_subfolders>
+ </file>
+ </files>
+ <fields>
+ <field>
+ <name>pr_number</name>
+ <length>-1</length>
+ <type>Integer</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <repeat>N</repeat>
+ <path>$.number</path>
+ </field>
+ <field>
+ <name>html_url</name>
+ <length>-1</length>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <repeat>N</repeat>
+ <path>$.html_url</path>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>736</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>FilterRows</type>
+ <name>pull fields ok?</name>
+ <compare>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>pr_number</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions>
+ <condition>
+ <negated>N</negated>
+ <operator>-</operator>
+ <leftvalue>pr_number</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>html_url</leftvalue>
+ <function>IS NOT NULL</function>
+ <conditions/>
+ </condition>
+ <condition>
+ <negated>N</negated>
+ <operator>AND</operator>
+ <leftvalue>html_url</leftvalue>
+ <function>STARTS WITH</function>
+ <value>
+ <name>constant</name>
+ <type>String</type>
+ <text>https://github.com/apache/hop-website/pull/</text>
+ <length>-1</length>
+ <precision>-1</precision>
+ <isnull>N</isnull>
+ <mask/>
+ </value>
+ <conditions/>
+ </condition>
+ </conditions>
+ </condition>
+ </compare>
+ <send_true_to>aggregate pull counts</send_true_to>
+ <send_false_to>Abort validation failed</send_false_to>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>928</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Dummy</type>
+ <name>success</name>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1696</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>GetVariable</type>
+ <name>Get variables</name>
+ <fields>
+ <field>
+ <name>min_pull_rows</name>
+ <variable>${PRM_MIN_PULL_ROWS}</variable>
+ <type>Integer</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <trim_type>none</trim_type>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>1312</xloc>
+ <yloc>272</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <order>
+ <hop>
+ <from>generate 1 row</from>
+ <to>GitHub request headers</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>GitHub request headers</from>
+ <to>fetch hop-website pull requests</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>fetch hop-website pull requests</from>
+ <to>parse pull JSON</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>parse pull JSON</from>
+ <to>pull fields ok?</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pull fields ok?</from>
+ <to>aggregate pull counts</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pull fields ok?</from>
+ <to>Abort validation failed</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pagination ok?</from>
+ <to>success</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>pagination ok?</from>
+ <to>Abort validation failed</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>aggregate pull counts</from>
+ <to>Get variables</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Get variables</from>
+ <to>pagination ok?</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <notepads>
+ <notepad>
+ <note>Integration test: live GET /repos/apache/hop-website/pulls?state=all&per_page=100 with LINK_HEADER pagination. No golden data.
+
+Passes when row_count >= ${PRM_MIN_PULL_ROWS} (default 250, ~3 pages of 100) and row_count equals distinct pr_number. Requires User-Agent. state=all is required (open-only would be far fewer rows).
+</note>
+ <fontname>SansSerif</fontname>
+ <fontsize>10</fontsize>
+ <fontbold>N</fontbold>
+ <fontitalic>N</fontitalic>
+ <fontcolorred>14</fontcolorred>
+ <fontcolorgreen>58</fontcolorgreen>
+ <fontcolorblue>90</fontcolorblue>
+ <backgroundcolorred>201</backgroundcolorred>
+ <backgroundcolorgreen>232</backgroundcolorgreen>
+ <backgroundcolorblue>251</backgroundcolorblue>
+ <bordercolorred>14</bordercolorred>
+ <bordercolorgreen>58</bordercolorgreen>
+ <bordercolorblue>90</bordercolorblue>
+ <xloc>64</xloc>
+ <yloc>32</yloc>
+ <width>720</width>
+ <height>200</height>
+ </notepad>
+ </notepads>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git a/integration-tests/transforms/main-0102-rest-client-github-pulls-link-header.hwf b/integration-tests/transforms/main-0102-rest-client-github-pulls-link-header.hwf
new file mode 100644
index 0000000000..d8b8d0633c
--- /dev/null
+++ b/integration-tests/transforms/main-0102-rest-client-github-pulls-link-header.hwf
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0102-rest-client-github-pulls-link-header</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>REST client LINK_HEADER pagination against live apache/hop-website pull requests.</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/06/06 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/06/06 12:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <DayOfMonth>1</DayOfMonth>
+ <hour>12</hour>
+ <intervalMinutes>60</intervalMinutes>
+ <intervalSeconds>0</intervalSeconds>
+ <minutes>0</minutes>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <weekDay>1</weekDay>
+ <parallel>N</parallel>
+ <xloc>208</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>0102-rest-client-github-pulls-link-header.hpl</name>
+ <description/>
+ <type>PIPELINE</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0102-rest-client-github-pulls-link-header.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <set_append_logfile>N</set_append_logfile>
+ <wait_until_finished>Y</wait_until_finished>
+ <create_parent_folder>N</create_parent_folder>
+ <run_configuration>local</run_configuration>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <parallel>N</parallel>
+ <xloc>448</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Success</name>
+ <description/>
+ <type>SUCCESS</type>
+ <attributes/>
+ <parallel>N</parallel>
+ <xloc>688</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Error</name>
+ <description/>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>N</always_log_rows>
+ <message>GitHub hop-website pull-request pagination test failed.</message>
+ <parallel>N</parallel>
+ <xloc>448</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Dummy</name>
+ <description/>
+ <type>DUMMY</type>
+ <attributes/>
+ <parallel>N</parallel>
+ <xloc>448</xloc>
+ <yloc>320</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>0102-rest-client-github-pulls-link-header.hpl</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>0102-rest-client-github-pulls-link-header.hpl</from>
+ <to>Success</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>0102-rest-client-github-pulls-link-header.hpl</from>
+ <to>Dummy</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Dummy</from>
+ <to>Error</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ </hops>
+ <notepads>
+ </notepads>
+ <attributes/>
+</workflow>
diff --git a/integration-tests/transforms/metadata/restconnection/github-hop-website.json b/integration-tests/transforms/metadata/restconnection/github-hop-website.json
new file mode 100644
index 0000000000..0930e7b211
--- /dev/null
+++ b/integration-tests/transforms/metadata/restconnection/github-hop-website.json
@@ -0,0 +1,10 @@
+{
+ "auth_type": "No Auth",
+ "base_url": "https://api.github.com",
+ "test_url": "https://api.github.com/repos/apache/hop-website/pulls?state=all&per_page=1",
+ "ignoreSsl": false,
+ "pagination_type": "LINK_HEADER",
+ "trustStorePassword": "Encrypted ",
+ "trustStoreFile": "",
+ "name": "github-hop-website"
+}
diff --git a/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java
index 19c7427c30..53faf5cd87 100644
--- a/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java
+++ b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java
@@ -22,6 +22,8 @@ import jakarta.ws.rs.client.ClientBuilder;
import jakarta.ws.rs.client.Invocation;
import jakarta.ws.rs.client.WebTarget;
import jakarta.ws.rs.core.HttpHeaders;
+import jakarta.ws.rs.core.MultivaluedHashMap;
+import jakarta.ws.rs.core.MultivaluedMap;
import jakarta.ws.rs.core.Response;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
@@ -105,7 +107,7 @@ public class RestConnection extends HopMetadataBase implements IHopMetadata {
private String password;
// Bearer auth
- @HopMetadataProperty(key = "bearer_token")
+ @HopMetadataProperty(key = "bearer_token", password = true)
private String bearerToken;
// API auth
@@ -134,6 +136,49 @@ public class RestConnection extends HopMetadataBase implements IHopMetadata {
@HopMetadataProperty(key = "certificateAlias")
private String certificateAlias;
+ /** Optional API pagination semantics (NONE by default). */
+ @HopMetadataProperty(key = "pagination_type")
+ private RestPaginationType paginationType = RestPaginationType.NONE;
+
+ /**
+ * Query parameter name used for cursor-based paging or page-number paging ({@link
+ * #paginationType}).
+ */
+ @HopMetadataProperty(key = "page_param_name")
+ private String pageParamName;
+
+ @HopMetadataProperty(key = "offset_param_name")
+ private String offsetParamName;
+
+ @HopMetadataProperty(key = "limit_param_name")
+ private String limitParamName;
+
+ /** Default page size used with {@link RestPaginationType#OFFSET_LIMIT} when not overridden. */
+ @HopMetadataProperty(key = "default_limit")
+ private int defaultLimit;
+
+ /** JsonPath against the JSON response body to read the next cursor token. */
+ @HopMetadataProperty(key = "cursor_json_path")
+ private String cursorJsonPath;
+
+ /** XPath against the XML response body to read the next cursor token. */
+ @HopMetadataProperty(key = "cursor_x_path")
+ private String cursorXPath;
+
+ /**
+ * JsonPath against the JSON response body to read the next page URL ({@link
+ * RestPaginationType#BODY_NEXT_URL}).
+ */
+ @HopMetadataProperty(key = "next_page_url_json_path")
+ private String nextPageUrlJsonPath;
+
+ /**
+ * XPath against the XML response body to read the next page URL ({@link
+ * RestPaginationType#BODY_NEXT_URL}).
+ */
+ @HopMetadataProperty(key = "next_page_url_x_path")
+ private String nextPageUrlXPath;
+
public RestConnection(IVariables variables) {
this.variables = variables;
}
@@ -192,30 +237,111 @@ public class RestConnection extends HopMetadataBase implements IHopMetadata {
}
}
- if (authType.equals(BASIC)
- && !StringUtils.isEmpty(username)
- && !StringUtils.isEmpty(password)) {
+ if (authTypeEquals(BASIC) && !StringUtils.isEmpty(username) && !StringUtils.isEmpty(password)) {
client.register(
HttpAuthenticationFeature.basic(
resolve(username), Encr.decryptPasswordOptionallyEncrypted(resolve(password))));
target = client.target(url);
invocationBuilder = target.request();
- } else if (authType.equals(API_KEY)) {
+ } else {
+ applyBearerAndApiKeyHeaders(invocationBuilder, null);
+ }
+ return invocationBuilder;
+ }
+
+ /**
+ * Applies API-key or Bearer auth to outbound headers (when {@code outboundHeaders} is non-null)
+ * or to {@code Invocation.Builder} via successive {@link Invocation.Builder#header} calls (when
+ * {@code outboundHeaders} is null). Skips when credentials are already on the map ({@code
+ * Authorization}, or the configured API-key header name). Intended for merging into maps before
+ * {@link Invocation.Builder#headers(javax.ws.rs.core.MultivaluedMap)}, which replaces Jersey's
+ * prior headers.
+ */
+ public void applyBearerAndApiKeyHeaders(
+ Invocation.Builder invocationBuilder, MultivaluedMap<String, Object> outboundHeaders) {
+ if (invocationBuilder == null || !supportsBearerOrApiKeyAuthHeaders()) {
+ return;
+ }
+ if (authHeadersAlreadyProvidedFromMap(outboundHeaders)) {
+ return;
+ }
+
+ MultivaluedMap<String, Object> authOnly =
+ outboundHeaders != null ? outboundHeaders : new MultivaluedHashMap<>();
+
+ if (authTypeEquals(API_KEY)) {
+ String headerName = resolve(authorizationHeaderName);
+ String headerValue =
+ Encr.decryptPasswordOptionallyEncrypted(resolve(authorizationHeaderValue));
+ if (Utils.isEmpty(headerName) || Utils.isEmpty(headerValue)) {
+ return;
+ }
if (!StringUtils.isEmpty(resolve(authorizationPrefix))) {
- invocationBuilder.header(
- resolve(authorizationHeaderName),
- resolve(authorizationPrefix)
- + " "
- + Encr.decryptPasswordOptionallyEncrypted(resolve(authorizationHeaderValue)));
+ authOnly.putSingle(headerName, resolve(authorizationPrefix) + " " + headerValue);
} else {
- invocationBuilder.header(
- resolve(authorizationHeaderName),
- Encr.decryptPasswordOptionallyEncrypted(resolve(authorizationHeaderValue)));
+ authOnly.putSingle(headerName, headerValue);
+ }
+ } else if (authTypeEquals(BEARER)) {
+ String token = Encr.decryptPasswordOptionallyEncrypted(resolve(bearerToken));
+ if (Utils.isEmpty(token)) {
+ return;
}
- } else if (authType.equals(BEARER) && !StringUtils.isEmpty(bearerToken)) {
- invocationBuilder.header(HttpHeaders.AUTHORIZATION, "Bearer " + resolve(bearerToken));
+ authOnly.putSingle(HttpHeaders.AUTHORIZATION, "Bearer " + token);
}
- return invocationBuilder;
+
+ if (outboundHeaders == null) {
+ authOnly.forEach(
+ (name, values) -> {
+ for (Object val : values) {
+ invocationBuilder.header(name, val);
+ }
+ });
+ }
+ }
+
+ /** True when this connection may add Bearer / API-key style headers before a request runs. */
+ private boolean supportsBearerOrApiKeyAuthHeaders() {
+ return authType != null && (authTypeEquals(API_KEY) || authTypeEquals(BEARER));
+ }
+
+ /**
+ * When merging into an outbound header map, skip adding connection auth if the map already
+ * supplies credentials: {@code Authorization}, or the connection's API-key header name (rows
+ * win).
+ */
+ private boolean authHeadersAlreadyProvidedFromMap(
+ MultivaluedMap<String, Object> outboundHeaders) {
+ if (outboundHeaders == null || outboundHeaders.isEmpty()) {
+ return false;
+ }
+ for (String key : outboundHeaders.keySet()) {
+ if (key == null) {
+ continue;
+ }
+ if ("Authorization".equalsIgnoreCase(key) && nonEmptyHeaderValue(outboundHeaders, key)) {
+ return true;
+ }
+ if (authTypeEquals(API_KEY)) {
+ String configuredName = resolve(authorizationHeaderName);
+ if (!Utils.isEmpty(configuredName)
+ && configuredName.equalsIgnoreCase(key)
+ && nonEmptyHeaderValue(outboundHeaders, key)) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ private boolean authTypeEquals(String canonicalLabel) {
+ return authType != null
+ && canonicalLabel != null
+ && authType.trim().equalsIgnoreCase(canonicalLabel.trim());
+ }
+
+ private static boolean nonEmptyHeaderValue(MultivaluedMap<String, Object> headers, String key) {
+ Object o = headers.getFirst(key);
+ return o != null && !Utils.isEmpty(o.toString().trim());
}
private void setProxyHost(String proxyHost, Integer proxyPort) {
@@ -238,8 +364,22 @@ public class RestConnection extends HopMetadataBase implements IHopMetadata {
return response;
}
+ /**
+ * Verifies connectivity using {@link #testUrl}. Mirrors the outbound header sequence used by the
+ * REST transform ({@link jakarta.ws.rs.client.Invocation.Builder#headers}: merge Bearer / API-key
+ * headers into the map before applying it); that avoids misleading green tests where only {@link
+ * jakarta.ws.rs.client.Invocation.Builder#header} ran and pagination later replaced headers
+ * without auth.
+ */
public void testConnection() throws HopException {
- Response response = getInvocationBuilder(resolve(testUrl)).get();
+ String url = resolve(testUrl);
+ Invocation.Builder invocationBuilder = getInvocationBuilder(url);
+ if (supportsBearerOrApiKeyAuthHeaders()) {
+ MultivaluedHashMap<String, Object> outbound = new MultivaluedHashMap<>();
+ applyBearerAndApiKeyHeaders(invocationBuilder, outbound);
+ invocationBuilder.headers(outbound);
+ }
+ Response response = invocationBuilder.get();
response.close();
}
diff --git a/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java
index 524bf959a4..dfa483b115 100644
--- a/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java
+++ b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java
@@ -20,12 +20,14 @@ package org.apache.hop.metadata.rest;
import java.util.Arrays;
import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.Const;
+import org.apache.hop.core.Props;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
import org.apache.hop.ui.core.dialog.ErrorDialog;
import org.apache.hop.ui.core.dialog.MessageBox;
+import org.apache.hop.ui.core.gui.GuiResource;
import org.apache.hop.ui.core.metadata.MetadataEditor;
import org.apache.hop.ui.core.metadata.MetadataManager;
import org.apache.hop.ui.core.widget.ComboVar;
@@ -33,6 +35,8 @@ import org.apache.hop.ui.core.widget.PasswordTextVar;
import org.apache.hop.ui.core.widget.TextVar;
import org.apache.hop.ui.hopgui.HopGui;
import org.eclipse.swt.SWT;
+import org.eclipse.swt.custom.CTabFolder;
+import org.eclipse.swt.custom.CTabItem;
import org.eclipse.swt.custom.ScrolledComposite;
import org.eclipse.swt.events.SelectionAdapter;
import org.eclipse.swt.events.SelectionEvent;
@@ -96,6 +100,16 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
private PasswordTextVar wKeyPassword;
private TextVar wCertificateAlias;
+ private ComboVar wPaginationType;
+ private TextVar wPageParamName;
+ private TextVar wOffsetParamName;
+ private TextVar wLimitParamName;
+ private TextVar wDefaultLimit;
+ private TextVar wCursorJsonPath;
+ private TextVar wCursorXPath;
+ private TextVar wNextPageUrlJsonPath;
+ private TextVar wNextPageUrlXPath;
+
private int middle;
private int margin;
private IVariables variables;
@@ -144,8 +158,31 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
wName.setLayoutData(fdName);
lastControl = wName;
- // start authentication group (now directly below name)
- gAuth = new Group(composite, SWT.SHADOW_ETCHED_IN);
+ CTabFolder wTabFolder = new CTabFolder(composite, SWT.BORDER);
+ PropsUi.setLook(wTabFolder, Props.WIDGET_STYLE_TAB);
+
+ FormData fdTabFolder = new FormData();
+ fdTabFolder.left = new FormAttachment(0, 0);
+ fdTabFolder.top = new FormAttachment(wName, margin);
+ fdTabFolder.right = new FormAttachment(100, 0);
+ fdTabFolder.bottom = new FormAttachment(100, -margin);
+ wTabFolder.setLayoutData(fdTabFolder);
+
+ CTabItem wConnectionTabItem = new CTabItem(wTabFolder, SWT.NONE);
+ wConnectionTabItem.setFont(GuiResource.getInstance().getFontDefault());
+ wConnectionTabItem.setText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Tab.Connection.Title"));
+
+ Composite wConnInner = new Composite(wTabFolder, SWT.NONE);
+ FormLayout connInnerLayout = new FormLayout();
+ connInnerLayout.marginWidth = PropsUi.getFormMargin();
+ connInnerLayout.marginHeight = PropsUi.getFormMargin();
+ wConnInner.setLayout(connInnerLayout);
+ PropsUi.setLook(wConnInner);
+ wConnectionTabItem.setControl(wConnInner);
+
+ // start authentication group (first section on Connection tab)
+ gAuth = new Group(wConnInner, SWT.SHADOW_ETCHED_IN);
gAuth.setText(BaseMessages.getString(PKG, "RestConnectionEditor.AuthGroup.Label"));
FormLayout gAuthLayout = new FormLayout();
gAuthLayout.marginWidth = 3;
@@ -215,12 +252,12 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
FormData fdAuth = new FormData();
fdAuth.left = new FormAttachment(0, 0);
- fdAuth.top = new FormAttachment(wName, margin);
+ fdAuth.top = new FormAttachment(0, margin);
fdAuth.right = new FormAttachment(100, 0);
gAuth.setLayoutData(fdAuth);
// start of URL group
- Group gUrl = new Group(composite, SWT.SHADOW_ETCHED_IN);
+ Group gUrl = new Group(wConnInner, SWT.SHADOW_ETCHED_IN);
gUrl.setText(BaseMessages.getString(PKG, "RestConnectionEditor.UrlGroup.Label"));
FormLayout gUrlLayout = new FormLayout();
gUrlLayout.marginWidth = 3;
@@ -270,7 +307,7 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
// end URL group
// start SSL group
- Group gSSLTrustStore = new Group(composite, SWT.SHADOW_ETCHED_IN);
+ Group gSSLTrustStore = new Group(wConnInner, SWT.SHADOW_ETCHED_IN);
gSSLTrustStore.setText(BaseMessages.getString(PKG, "RestConnectionEditor.SSLGroup.Label"));
FormLayout gSSLTrustStoreLayout = new FormLayout();
gSSLTrustStoreLayout.marginWidth = 3;
@@ -500,9 +537,26 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
fdSSLTrustStore.left = new FormAttachment(0, 0);
fdSSLTrustStore.right = new FormAttachment(100, 0);
fdSSLTrustStore.top = new FormAttachment(gUrl, margin);
+ fdSSLTrustStore.bottom = new FormAttachment(100, -margin);
gSSLTrustStore.setLayoutData(fdSSLTrustStore);
// end SSL group
+ CTabItem wPaginationTabItem = new CTabItem(wTabFolder, SWT.NONE);
+ wPaginationTabItem.setFont(GuiResource.getInstance().getFontDefault());
+ wPaginationTabItem.setText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Tab.Pagination.Title"));
+
+ Composite wPagInner = new Composite(wTabFolder, SWT.NONE);
+ FormLayout pagLayout = new FormLayout();
+ pagLayout.marginWidth = PropsUi.getFormMargin();
+ pagLayout.marginHeight = PropsUi.getFormMargin();
+ wPagInner.setLayout(pagLayout);
+ PropsUi.setLook(wPagInner);
+ buildPaginationWidgets(wPagInner);
+ wPaginationTabItem.setControl(wPagInner);
+
+ wTabFolder.setSelection(wConnectionTabItem);
+
setWidgetsContent();
wsAuthComp.setExpandHorizontal(true);
@@ -522,11 +576,255 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
wKeyStorePassword,
wKeyStoreType,
wKeyPassword,
- wCertificateAlias
+ wCertificateAlias,
+ wPaginationType,
+ wPageParamName,
+ wOffsetParamName,
+ wLimitParamName,
+ wDefaultLimit,
+ wCursorJsonPath,
+ wCursorXPath,
+ wNextPageUrlJsonPath,
+ wNextPageUrlXPath
};
enableControls(controls);
}
+ /** Builds paging settings on the dedicated Pagination tab. */
+ private void buildPaginationWidgets(Composite p) {
+ Label wlInfo = new Label(p, SWT.LEFT | SWT.WRAP);
+ wlInfo.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.Info"));
+ PropsUi.setLook(wlInfo);
+ FormData fdi = new FormData();
+ fdi.left = new FormAttachment(0, margin);
+ fdi.right = new FormAttachment(100, -margin);
+ fdi.top = new FormAttachment(0, margin);
+ wlInfo.setLayoutData(fdi);
+
+ Control lastControl = wlInfo;
+
+ Label wlPaginationType = new Label(p, SWT.RIGHT);
+ wlPaginationType.setText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.Type.Label"));
+ PropsUi.setLook(wlPaginationType);
+ FormData fdlType = new FormData();
+ fdlType.top = new FormAttachment(lastControl, margin);
+ fdlType.left = new FormAttachment(0, margin);
+ fdlType.right = new FormAttachment(middle, -margin);
+ wlPaginationType.setLayoutData(fdlType);
+
+ wPaginationType = new ComboVar(variables, p, SWT.READ_ONLY | SWT.BORDER);
+ PropsUi.setLook(wPaginationType);
+ wPaginationType.setItems(
+ Arrays.stream(RestPaginationType.values()).map(Enum::name).toArray(String[]::new));
+ FormData fdType = new FormData();
+ fdType.top = new FormAttachment(wlPaginationType, 0, SWT.CENTER);
+ fdType.left = new FormAttachment(middle, margin);
+ fdType.right = new FormAttachment(100, -margin);
+ wPaginationType.setLayoutData(fdType);
+ wPaginationType.addListener(
+ SWT.Selection,
+ e -> {
+ markChangedIfUserEdit();
+ refreshPaginationSensitiveFields();
+ });
+ lastControl = wPaginationType;
+
+ Label wlPage = new Label(p, SWT.RIGHT);
+ wlPage.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.PageParam.Label"));
+ PropsUi.setLook(wlPage);
+ FormData fdLP = new FormData();
+ fdLP.top = new FormAttachment(lastControl, margin);
+ fdLP.left = new FormAttachment(0, margin);
+ fdLP.right = new FormAttachment(middle, -margin);
+ wlPage.setLayoutData(fdLP);
+
+ wPageParamName = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wPageParamName);
+ wPageParamName.setToolTipText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.PageParam.Tooltip"));
+ FormData fdP = new FormData();
+ fdP.top = new FormAttachment(wlPage, 0, SWT.CENTER);
+ fdP.left = new FormAttachment(middle, margin);
+ fdP.right = new FormAttachment(100, -margin);
+ wPageParamName.setLayoutData(fdP);
+ lastControl = wPageParamName;
+
+ Label wlOff = new Label(p, SWT.RIGHT);
+ wlOff.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.OffsetParam.Label"));
+ PropsUi.setLook(wlOff);
+ FormData fdLOff = new FormData();
+ fdLOff.top = new FormAttachment(lastControl, margin);
+ fdLOff.left = new FormAttachment(0, margin);
+ fdLOff.right = new FormAttachment(middle, -margin);
+ wlOff.setLayoutData(fdLOff);
+
+ wOffsetParamName = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wOffsetParamName);
+ FormData fdOff = new FormData();
+ fdOff.top = new FormAttachment(wlOff, 0, SWT.CENTER);
+ fdOff.left = new FormAttachment(middle, margin);
+ fdOff.right = new FormAttachment(100, -margin);
+ wOffsetParamName.setLayoutData(fdOff);
+ lastControl = wOffsetParamName;
+
+ Label wlLim = new Label(p, SWT.RIGHT);
+ wlLim.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.LimitParam.Label"));
+ PropsUi.setLook(wlLim);
+ FormData fdLL = new FormData();
+ fdLL.top = new FormAttachment(lastControl, margin);
+ fdLL.left = new FormAttachment(0, margin);
+ fdLL.right = new FormAttachment(middle, -margin);
+ wlLim.setLayoutData(fdLL);
+
+ wLimitParamName = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wLimitParamName);
+ FormData fdL = new FormData();
+ fdL.top = new FormAttachment(wlLim, 0, SWT.CENTER);
+ fdL.left = new FormAttachment(middle, margin);
+ fdL.right = new FormAttachment(100, -margin);
+ wLimitParamName.setLayoutData(fdL);
+ lastControl = wLimitParamName;
+
+ Label wlDef = new Label(p, SWT.RIGHT);
+ wlDef.setText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.DefaultLimit.Label"));
+ PropsUi.setLook(wlDef);
+ wlDef.setToolTipText(
+ BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.DefaultLimit.Tooltip"));
+ FormData fdLD = new FormData();
+ fdLD.top = new FormAttachment(lastControl, margin);
+ fdLD.left = new FormAttachment(0, margin);
+ fdLD.right = new FormAttachment(middle, -margin);
+ wlDef.setLayoutData(fdLD);
+
+ wDefaultLimit = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wDefaultLimit);
+ wDefaultLimit.setToolTipText(wlDef.getToolTipText());
+ FormData fdDef = new FormData();
+ fdDef.top = new FormAttachment(wlDef, 0, SWT.CENTER);
+ fdDef.left = new FormAttachment(middle, margin);
+ fdDef.right = new FormAttachment(100, -margin);
+ wDefaultLimit.setLayoutData(fdDef);
+ lastControl = wDefaultLimit;
+
+ Label wlCj = new Label(p, SWT.RIGHT);
+ wlCj.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.CursorJson.Label"));
+ PropsUi.setLook(wlCj);
+ FormData fdLCj = new FormData();
+ fdLCj.top = new FormAttachment(lastControl, margin);
+ fdLCj.left = new FormAttachment(0, margin);
+ fdLCj.right = new FormAttachment(middle, -margin);
+ wlCj.setLayoutData(fdLCj);
+
+ wCursorJsonPath = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wCursorJsonPath);
+ FormData fdCj = new FormData();
+ fdCj.top = new FormAttachment(wlCj, 0, SWT.CENTER);
+ fdCj.left = new FormAttachment(middle, margin);
+ fdCj.right = new FormAttachment(100, -margin);
+ wCursorJsonPath.setLayoutData(fdCj);
+ lastControl = wCursorJsonPath;
+
+ Label wlCx = new Label(p, SWT.RIGHT);
+ wlCx.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.CursorXPath.Label"));
+ PropsUi.setLook(wlCx);
+ FormData fdLX = new FormData();
+ fdLX.top = new FormAttachment(lastControl, margin);
+ fdLX.left = new FormAttachment(0, margin);
+ fdLX.right = new FormAttachment(middle, -margin);
+ wlCx.setLayoutData(fdLX);
+
+ wCursorXPath = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wCursorXPath);
+ FormData fdX = new FormData();
+ fdX.top = new FormAttachment(wlCx, 0, SWT.CENTER);
+ fdX.left = new FormAttachment(middle, margin);
+ fdX.right = new FormAttachment(100, -margin);
+ wCursorXPath.setLayoutData(fdX);
+ lastControl = wCursorXPath;
+
+ Label wlNu = new Label(p, SWT.RIGHT);
+ wlNu.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.NextUrlJson.Label"));
+ PropsUi.setLook(wlNu);
+ FormData fdLNu = new FormData();
+ fdLNu.top = new FormAttachment(lastControl, margin);
+ fdLNu.left = new FormAttachment(0, margin);
+ fdLNu.right = new FormAttachment(middle, -margin);
+ wlNu.setLayoutData(fdLNu);
+
+ wNextPageUrlJsonPath = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wNextPageUrlJsonPath);
+ FormData fdNu = new FormData();
+ fdNu.top = new FormAttachment(wlNu, 0, SWT.CENTER);
+ fdNu.left = new FormAttachment(middle, margin);
+ fdNu.right = new FormAttachment(100, -margin);
+ wNextPageUrlJsonPath.setLayoutData(fdNu);
+ lastControl = wNextPageUrlJsonPath;
+
+ Label wlNx = new Label(p, SWT.RIGHT);
+ wlNx.setText(BaseMessages.getString(PKG, "RestConnectionEditor.Pagination.NextUrlXPath.Label"));
+ PropsUi.setLook(wlNx);
+ FormData fdLNx = new FormData();
+ fdLNx.top = new FormAttachment(lastControl, margin);
+ fdLNx.left = new FormAttachment(0, margin);
+ fdLNx.right = new FormAttachment(middle, -margin);
+ wlNx.setLayoutData(fdLNx);
+
+ wNextPageUrlXPath = new TextVar(variables, p, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wNextPageUrlXPath);
+ FormData fdNx = new FormData();
+ fdNx.top = new FormAttachment(wlNx, 0, SWT.CENTER);
+ fdNx.left = new FormAttachment(middle, margin);
+ fdNx.right = new FormAttachment(100, -margin);
+ wNextPageUrlXPath.setLayoutData(fdNx);
+
+ refreshPaginationSensitiveFields();
+ }
+
+ private void refreshPaginationSensitiveFields() {
+ if (wPaginationType == null || wPaginationType.isDisposed()) {
+ return;
+ }
+ RestPaginationType t;
+ try {
+ t = RestPaginationType.valueOf(wPaginationType.getText());
+ } catch (IllegalArgumentException ex) {
+ t = RestPaginationType.NONE;
+ }
+ boolean bodyCursor = RestPaginationType.BODY_CURSOR.equals(t);
+ boolean headerCursor = RestPaginationType.HEADER_CURSOR.equals(t);
+ boolean bodyNextUrl = RestPaginationType.BODY_NEXT_URL.equals(t);
+ boolean pageOrCursor =
+ RestPaginationType.PAGE_NUMBER.equals(t)
+ || RestPaginationType.CURSOR.equals(t)
+ || bodyCursor
+ || headerCursor;
+ boolean offset = RestPaginationType.OFFSET_LIMIT.equals(t);
+ boolean cursor = RestPaginationType.CURSOR.equals(t) || bodyCursor || headerCursor;
+ boolean limitBatch = offset || bodyCursor || headerCursor;
+
+ wPageParamName.setEnabled(pageOrCursor);
+ wOffsetParamName.setEnabled(offset);
+ wLimitParamName.setEnabled(limitBatch);
+ wDefaultLimit.setEnabled(limitBatch);
+ wCursorJsonPath.setEnabled(cursor);
+ wCursorXPath.setEnabled(cursor);
+ wNextPageUrlJsonPath.setEnabled(bodyNextUrl);
+ wNextPageUrlXPath.setEnabled(bodyNextUrl);
+ }
+
+ private static int parseUnsignedIntSafe(String text) {
+ if (StringUtils.isEmpty(Const.trim(text))) {
+ return 0;
+ }
+ try {
+ return Math.max(0, Integer.parseInt(Const.trim(text)));
+ } catch (NumberFormatException e) {
+ return 0;
+ }
+ }
+
private void enableControls(Control[] controls) {
for (Control control : controls) {
if (control == null || control.isDisposed()) {
@@ -859,6 +1157,31 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
// ignore
}
}
+
+ RestPaginationType pag =
+ metadata.getPaginationType() == null
+ ? RestPaginationType.NONE
+ : metadata.getPaginationType();
+ String[] items = wPaginationType.getItems();
+ int idx = Arrays.asList(items).indexOf(pag.name());
+ if (idx >= 0) {
+ wPaginationType.select(idx);
+ }
+
+ wPageParamName.setText(Const.NVL(metadata.getPageParamName(), ""));
+ wOffsetParamName.setText(Const.NVL(metadata.getOffsetParamName(), ""));
+ wLimitParamName.setText(Const.NVL(metadata.getLimitParamName(), ""));
+ if (metadata.getDefaultLimit() > 0) {
+ wDefaultLimit.setText(Integer.toString(metadata.getDefaultLimit()));
+ } else {
+ wDefaultLimit.setText("");
+ }
+ wCursorJsonPath.setText(Const.NVL(metadata.getCursorJsonPath(), ""));
+ wCursorXPath.setText(Const.NVL(metadata.getCursorXPath(), ""));
+ wNextPageUrlJsonPath.setText(Const.NVL(metadata.getNextPageUrlJsonPath(), ""));
+ wNextPageUrlXPath.setText(Const.NVL(metadata.getNextPageUrlXPath(), ""));
+
+ refreshPaginationSensitiveFields();
} finally {
loadingContent = false;
}
@@ -893,6 +1216,22 @@ public class RestConnectionEditor extends MetadataEditor<RestConnection> {
connection.setAuthorizationHeaderValue(wAuthorizationValue.getText());
}
// Note: Certificate auth doesn't have additional fields in auth section
+
+ RestPaginationType pType;
+ try {
+ pType = RestPaginationType.valueOf(wPaginationType.getText());
+ } catch (IllegalArgumentException ex) {
+ pType = RestPaginationType.NONE;
+ }
+ connection.setPaginationType(pType);
+ connection.setPageParamName(wPageParamName.getText());
+ connection.setOffsetParamName(wOffsetParamName.getText());
+ connection.setLimitParamName(wLimitParamName.getText());
+ connection.setDefaultLimit(parseUnsignedIntSafe(wDefaultLimit.getText()));
+ connection.setCursorJsonPath(wCursorJsonPath.getText());
+ connection.setCursorXPath(wCursorXPath.getText());
+ connection.setNextPageUrlJsonPath(wNextPageUrlJsonPath.getText());
+ connection.setNextPageUrlXPath(wNextPageUrlXPath.getText());
}
@Override
diff --git a/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestPaginationType.java b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestPaginationType.java
new file mode 100644
index 0000000000..fb6c33582e
--- /dev/null
+++ b/plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestPaginationType.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.metadata.rest;
+
+/**
+ * Describes how pagination tokens are propagated for REST requests configured on {@link
+ * RestConnection}.
+ */
+public enum RestPaginationType {
+ /** No pagination (default). */
+ NONE,
+
+ /**
+ * Use the RFC 5988 {@code Link} response header: follow the bracketed URI whose {@code rel}
+ * relation type includes {@code next}. Multiple header fields and comma-separated link-values are
+ * supported ({@code prev} and others are ignored unless they advertise {@code next}).
+ */
+ LINK_HEADER,
+
+ /** Encode {@code offset} and {@code limit} query parameters (names from metadata). */
+ OFFSET_LIMIT,
+
+ /**
+ * Encode a {@code page} (or similarly named) query parameter with an incrementing page index,
+ * starting at 1.
+ */
+ PAGE_NUMBER,
+
+ /**
+ * Encode a cursor token query parameter ({@link RestConnection#pageParamName}, typically named
+ * {@code cursor}) from the previous response body using JsonPath/XPath expressions in metadata.
+ */
+ CURSOR,
+
+ /**
+ * Merge cursor and batch-size fields into the HTTP request body on POST/PUT/PATCH (form
+ * url-encoded or JSON). The next cursor is read from the response with JsonPath/XPath (as with
+ * {@link #CURSOR}). Pagination stops when the next cursor is empty — not when a page returns zero
+ * rows (Slack-style APIs may return sparse pages while {@code next_cursor} is still set).
+ *
+ * <p>On GET requests, cursor and batch-size are sent as query parameters instead (HTTP clients
+ * typically ignore GET bodies).
+ */
+ BODY_CURSOR,
+
+ /**
+ * Send the cursor token as an HTTP request header ({@link RestConnection#pageParamName}). The
+ * next cursor is read from the response with JsonPath/XPath. Pagination stops when the next
+ * cursor is empty.
+ */
+ HEADER_CURSOR,
+
+ /**
+ * Read the next page URL from the JSON/XML response body (JsonPath/XPath on the connection) and
+ * request it as an absolute URL on the following iteration (same loop semantics as {@link
+ * #LINK_HEADER}).
+ */
+ BODY_NEXT_URL
+}
diff --git a/plugins/misc/rest/src/main/resources/org/apache/hop/metadata/rest/messages/messages_en_US.properties b/plugins/misc/rest/src/main/resources/org/apache/hop/metadata/rest/messages/messages_en_US.properties
index 3bb19b33a0..b4bd03ee5c 100644
--- a/plugins/misc/rest/src/main/resources/org/apache/hop/metadata/rest/messages/messages_en_US.properties
+++ b/plugins/misc/rest/src/main/resources/org/apache/hop/metadata/rest/messages/messages_en_US.properties
@@ -43,4 +43,18 @@ RestConnectionEditor.KeyPassword.Tooltip=Optional: Separate password for private
RestConnectionEditor.CertificateAlias.Label=Certificate alias
RestConnectionEditor.CertificateAlias.Tooltip=Optional: Alias of certificate in keystore (if multiple certificates exist)
RestConnectionEditor.CertificateFiles=Certificate Files (*.p12, *.pfx, *.jks)
-RestConnectionEditor.Certificate.Info=Certificate authentication (mTLS) is configured in the SSL section above. Provide your client certificate keystore file, password, and type.
\ No newline at end of file
+RestConnectionEditor.Certificate.Info=Certificate authentication (mTLS) is configured in the SSL section above. Provide your client certificate keystore file, password, and type.
+RestConnectionEditor.Tab.Connection.Title=Connection
+RestConnectionEditor.Tab.Pagination.Title=Pagination
+RestConnectionEditor.Pagination.Info=Choose how paging tokens propagate for REST transforms that enable pagination. OFFSET_LIMIT and PAGE_NUMBER use query parameters. LINK_HEADER follows RFC 5988 rel=\"next\". CURSOR uses query parameters. BODY_CURSOR merges cursor and batch-size into the request body (GET falls back to query). HEADER_CURSOR sends the cursor as a request header. BODY_NEXT_URL follows a next-page URL from the response body. Cursor-style modes stop when the next cursor is empty.
+RestConnectionEditor.Pagination.Type.Label=Pagination strategy
+RestConnectionEditor.Pagination.PageParam.Label=Page / cursor parameter name
+RestConnectionEditor.Pagination.PageParam.Tooltip=Query parameter for PAGE_NUMBER or CURSOR (e.g. page, cursor). Body field name for BODY_CURSOR (e.g. cursor). Request header name for HEADER_CURSOR. Ignored by OFFSET_LIMIT, LINK_HEADER, and BODY_NEXT_URL.
+RestConnectionEditor.Pagination.OffsetParam.Label=Offset query parameter name (OFFSET_LIMIT)
+RestConnectionEditor.Pagination.LimitParam.Label=Limit / batch-size parameter name
+RestConnectionEditor.Pagination.DefaultLimit.Label=Default batch size (0 = Hop default fallback)
+RestConnectionEditor.Pagination.DefaultLimit.Tooltip=Page size for OFFSET_LIMIT (query) or BODY_CURSOR (request body). When 0, Hop falls back to 100 for BODY_CURSOR.
+RestConnectionEditor.Pagination.CursorJson.Label=Cursor JsonPath (CURSOR / BODY_CURSOR / HEADER_CURSOR + JSON)
+RestConnectionEditor.Pagination.CursorXPath.Label=Cursor XPath (CURSOR / BODY_CURSOR / HEADER_CURSOR + XML)
+RestConnectionEditor.Pagination.NextUrlJson.Label=Next page URL JsonPath (BODY_NEXT_URL + JSON)
+RestConnectionEditor.Pagination.NextUrlXPath.Label=Next page URL XPath (BODY_NEXT_URL + XML)
\ No newline at end of file
diff --git a/plugins/transforms/json/pom.xml b/plugins/transforms/json/pom.xml
index 2eb647cccf..12d1419440 100644
--- a/plugins/transforms/json/pom.xml
+++ b/plugins/transforms/json/pom.xml
@@ -31,7 +31,6 @@
<properties>
<encoder.version>1.4.0</encoder.version>
- <json-path.version>2.10.0</json-path.version>
</properties>
<dependencyManagement>
diff --git a/plugins/transforms/rest/pom.xml b/plugins/transforms/rest/pom.xml
index 0b8261e943..155c807844 100644
--- a/plugins/transforms/rest/pom.xml
+++ b/plugins/transforms/rest/pom.xml
@@ -30,6 +30,11 @@
<name>Hop Plugins Transforms REST</name>
<dependencies>
+ <dependency>
+ <groupId>com.jayway.jsonpath</groupId>
+ <artifactId>json-path</artifactId>
+ <version>${json-path.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.hop</groupId>
<artifactId>hop-misc-rest</artifactId>
diff --git a/plugins/transforms/rest/src/assembly/assembly.xml b/plugins/transforms/rest/src/assembly/assembly.xml
index 86d8fd80cf..ff0bae5635 100644
--- a/plugins/transforms/rest/src/assembly/assembly.xml
+++ b/plugins/transforms/rest/src/assembly/assembly.xml
@@ -50,6 +50,12 @@
</includes>
<outputDirectory>plugins/transforms/rest</outputDirectory>
</dependencySet>
+ <dependencySet>
+ <includes>
+ <include>com.jayway.jsonpath:json-path:jar</include>
+ </includes>
+ <outputDirectory>plugins/transforms/rest/lib</outputDirectory>
+ </dependencySet>
<dependencySet>
<includes>
<include>org.glassfish.jersey.connectors:jersey-apache5-connector:jar</include>
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPaging.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPaging.java
new file mode 100644
index 0000000000..9b9dfd33fe
--- /dev/null
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPaging.java
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import jakarta.ws.rs.core.MultivaluedMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.StringJoiner;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.hop.core.util.Utils;
+
+/**
+ * RFC 5988 / RFC 8288 {@code Link} header handling for pagination: find the bracketed URI whose
+ * {@code rel=} parameter mentions the {@code next} relation token (as used by GitHub, Shopify, and
+ * other REST APIs that advertise the next page in the {@code Link} header rather than the body).
+ *
+ * @see <a href="https://www.rfc-editor.org/rfc/rfc5988">RFC 5988 Web Linking</a>
+ * @see <a href="https://www.rfc-editor.org/rfc/rfc8288">RFC 8288</a>
+ * @see <a href="https://docs.github.com/rest/guides/using-pagination-in-the-rest-api">GitHub
+ * pagination</a>
+ */
+final class LinkHeaderPaging {
+
+ /**
+ * Parameters after closing {@code >} until end of segment, e.g. {@code ; rel=\"next\"; type=}.
+ */
+ private static final Pattern URI_REF =
+ Pattern.compile("^\\s*<([^>]*)>\\s*(.*)\\s*", Pattern.DOTALL);
+
+ private static final Pattern REL_PARAM =
+ Pattern.compile(
+ "(?:^|[;,])\\s*rel\\s*=\\s*(?:\"([^\"]*)\"|'([^']*)'|([^;,\\s]+))",
+ Pattern.CASE_INSENSITIVE);
+
+ private LinkHeaderPaging() {}
+
+ /** Collects {@code Link} header values case-insensitively and merges them comma-wise. */
+ static String concatenateLinkHeaders(MultivaluedMap<String, ?> headers) {
+ if (headers == null) {
+ return "";
+ }
+ StringJoiner joiner = new StringJoiner(", ");
+ for (Map.Entry<String, ? extends List<?>> e : headers.entrySet()) {
+ if (e.getKey() != null && "link".equalsIgnoreCase(e.getKey()) && e.getValue() != null) {
+ for (Object o : e.getValue()) {
+ if (o != null) {
+ String s = o.toString().trim();
+ if (!s.isEmpty()) {
+ joiner.add(s);
+ }
+ }
+ }
+ }
+ }
+ return joiner.toString().trim();
+ }
+
+ /**
+ * @return bracket-URI whose {@code rel} includes the {@code next} token, else null
+ */
+ static String findFirstUriWithRelNext(MultivaluedMap<String, ?> headers) {
+ String concatenated = concatenateLinkHeaders(headers);
+ if (Utils.isEmpty(concatenated)) {
+ return null;
+ }
+ for (String segment : splitIntoLinkSegments(concatenated)) {
+ Matcher mUri = URI_REF.matcher(segment);
+ if (!mUri.matches()) {
+ continue;
+ }
+ String uri = mUri.group(1).trim();
+ String tail = mUri.group(2);
+ if (!uri.isEmpty() && relationsIncludeNext(tail)) {
+ return uri;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Split a {@code Link} field value between link-values. Commas appear inside URIs rarely; commas
+ * inside {@code <>} are skipped so query strings with commas do not terminate a segment wrongly.
+ */
+ static List<String> splitIntoLinkSegments(String concatenatedValues) {
+ List<String> out = new ArrayList<>();
+ boolean inAngles = false;
+ int segStart = 0;
+ for (int i = 0; i <= concatenatedValues.length(); i++) {
+ if (i < concatenatedValues.length()) {
+ char c = concatenatedValues.charAt(i);
+ if (c == '<') {
+ inAngles = true;
+ } else if (c == '>') {
+ inAngles = false;
+ }
+ }
+ if (i == concatenatedValues.length() || (concatenatedValues.charAt(i) == ',' && !inAngles)) {
+ String segment = concatenatedValues.substring(segStart, i).trim();
+ if (!segment.isEmpty()) {
+ out.add(segment);
+ }
+ segStart = i + 1;
+ }
+ }
+ return out;
+ }
+
+ static boolean relationsIncludeNext(String segmentTail) {
+ if (segmentTail == null || segmentTail.isEmpty()) {
+ return false;
+ }
+ Matcher m = REL_PARAM.matcher(segmentTail);
+ while (m.find()) {
+ String v = nonEmptyGroup(m.group(1), nonEmptyGroup(m.group(2), m.group(3)));
+ if (v == null || v.isEmpty()) {
+ continue;
+ }
+ for (String token : v.trim().split("\\s+")) {
+ if (!token.isEmpty() && "next".equalsIgnoreCase(token)) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ private static String nonEmptyGroup(String first, String second) {
+ if (first != null && !first.isEmpty()) {
+ return first;
+ }
+ return second;
+ }
+}
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMerge.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMerge.java
new file mode 100644
index 0000000000..2278cb0de0
--- /dev/null
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMerge.java
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import jakarta.ws.rs.core.MediaType;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.hop.core.util.Utils;
+
+/** Merges paging tokens into HTTP request bodies for {@link RestPaginationType#BODY_CURSOR}. */
+final class PagingBodyMerge {
+
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private PagingBodyMerge() {}
+
+ static String merge(
+ String entityString,
+ LinkedHashMap<String, String> pagingBodyParams,
+ String contentTypeHeader) {
+ if (pagingBodyParams == null || pagingBodyParams.isEmpty()) {
+ return entityString;
+ }
+ if (isJsonContentType(contentTypeHeader)) {
+ return mergeJsonBody(entityString, pagingBodyParams);
+ }
+ return mergeFormUrlEncodedBody(entityString, pagingBodyParams);
+ }
+
+ static boolean isJsonContentType(String contentTypeHeader) {
+ if (Utils.isEmpty(contentTypeHeader)) {
+ return false;
+ }
+ try {
+ return MediaType.APPLICATION_JSON_TYPE.isCompatible(MediaType.valueOf(contentTypeHeader));
+ } catch (IllegalArgumentException ex) {
+ return contentTypeHeader.toLowerCase().contains("json");
+ }
+ }
+
+ static boolean isFormUrlEncodedContentType(String contentTypeHeader) {
+ if (Utils.isEmpty(contentTypeHeader)) {
+ return false;
+ }
+ try {
+ return MediaType.APPLICATION_FORM_URLENCODED_TYPE.isCompatible(
+ MediaType.valueOf(contentTypeHeader));
+ } catch (IllegalArgumentException ex) {
+ return contentTypeHeader.toLowerCase().contains("x-www-form-urlencoded");
+ }
+ }
+
+ static LinkedHashMap<String, String> parseFormUrlEncoded(String body) {
+ LinkedHashMap<String, String> map = new LinkedHashMap<>();
+ if (Utils.isEmpty(body)) {
+ return map;
+ }
+ for (String pair : body.split("&")) {
+ if (pair.isEmpty()) {
+ continue;
+ }
+ int eq = pair.indexOf('=');
+ if (eq < 0) {
+ map.put(decodeFormComponent(pair), "");
+ } else {
+ map.put(
+ decodeFormComponent(pair.substring(0, eq)),
+ decodeFormComponent(pair.substring(eq + 1)));
+ }
+ }
+ return map;
+ }
+
+ static String mergeFormUrlEncodedBody(
+ String entityString, LinkedHashMap<String, String> pagingBodyParams) {
+ LinkedHashMap<String, String> merged = parseFormUrlEncoded(entityString);
+ merged.putAll(pagingBodyParams);
+ StringBuilder sb = new StringBuilder();
+ for (Map.Entry<String, String> e : merged.entrySet()) {
+ if (Utils.isEmpty(e.getKey())) {
+ continue;
+ }
+ if (!sb.isEmpty()) {
+ sb.append('&');
+ }
+ sb.append(encodeFormComponent(e.getKey()));
+ sb.append('=');
+ sb.append(encodeFormComponent(e.getValue() == null ? "" : e.getValue()));
+ }
+ return sb.toString();
+ }
+
+ static String mergeJsonBody(String entityString, LinkedHashMap<String, String> pagingBodyParams) {
+ try {
+ ObjectNode node;
+ if (Utils.isEmpty(entityString)) {
+ node = MAPPER.createObjectNode();
+ } else {
+ JsonNode parsed = MAPPER.readTree(entityString);
+ if (parsed instanceof ObjectNode objectNode) {
+ node = objectNode;
+ } else {
+ node = MAPPER.createObjectNode();
+ }
+ }
+ for (Map.Entry<String, String> e : pagingBodyParams.entrySet()) {
+ if (Utils.isEmpty(e.getKey())) {
+ continue;
+ }
+ String value = e.getValue() == null ? "" : e.getValue();
+ if (value.matches("-?\\d+")) {
+ try {
+ node.put(e.getKey(), Long.parseLong(value));
+ continue;
+ } catch (NumberFormatException ignored) {
+ // fall through to string
+ }
+ }
+ node.put(e.getKey(), value);
+ }
+ return MAPPER.writeValueAsString(node);
+ } catch (Exception ex) {
+ return entityString;
+ }
+ }
+
+ private static String decodeFormComponent(String value) {
+ return URLDecoder.decode(value, StandardCharsets.UTF_8);
+ }
+
+ private static String encodeFormComponent(String value) {
+ return URLEncoder.encode(value, StandardCharsets.UTF_8);
+ }
+}
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java
index e58322bbcb..caf1d04d95 100644
--- a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java
@@ -19,6 +19,10 @@ package org.apache.hop.pipeline.transforms.rest;
import static org.apache.hop.core.Const.NVL;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.jayway.jsonpath.Configuration;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.Option;
import jakarta.ws.rs.ProcessingException;
import jakarta.ws.rs.client.Client;
import jakarta.ws.rs.client.ClientBuilder;
@@ -34,17 +38,36 @@ import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.net.URI;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Supplier;
import javax.net.ssl.SSLContext;
+import javax.xml.XMLConstants;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import javax.xml.xpath.XPath;
+import javax.xml.xpath.XPathConstants;
+import javax.xml.xpath.XPathExpression;
+import javax.xml.xpath.XPathFactory;
import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.Const;
import org.apache.hop.core.encryption.Encr;
@@ -52,12 +75,14 @@ import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopRuntimeException;
import org.apache.hop.core.row.RowDataUtil;
import org.apache.hop.core.util.HttpClientManager;
+import org.apache.hop.core.util.StringUtil;
import org.apache.hop.core.util.Utils;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.lineage.LineageHttpIoEmitter;
import org.apache.hop.lineage.model.HttpDirection;
import org.apache.hop.lineage.model.HttpLineagePayload;
import org.apache.hop.metadata.rest.RestConnection;
+import org.apache.hop.metadata.rest.RestPaginationType;
import org.apache.hop.pipeline.Pipeline;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.BaseTransform;
@@ -71,6 +96,9 @@ import org.glassfish.jersey.client.RequestEntityProcessing;
import org.glassfish.jersey.client.authentication.HttpAuthenticationFeature;
import org.glassfish.jersey.uri.UriComponent;
import org.json.simple.JSONObject;
+import org.w3c.dom.Document;
+import org.w3c.dom.NodeList;
+import org.xml.sax.InputSource;
public class Rest extends BaseTransform<RestMeta, RestData> {
private static final Class<?> PKG = RestMeta.class;
@@ -79,6 +107,55 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
private String baseUrl = "";
private RestConnection connection;
+ private static final Configuration JSON_PATH_CONFIGURATION =
+ Configuration.builder()
+ .options(Option.DEFAULT_PATH_LEAF_TO_NULL, Option.SUPPRESS_EXCEPTIONS)
+ .build();
+
+ private final ObjectMapper paginationJsonMapper = new ObjectMapper();
+
+ /**
+ * Outcome of one HTTP invocation (body, timing, serialization of headers needed for paging and
+ * output fields).
+ */
+ protected static final class RestExchangeResult {
+ final String body;
+ final int status;
+ final long responseTimeMs;
+ final String headerJson;
+ final MultivaluedMap<String, Object> headers;
+
+ /** Effective request URL for this exchange (after paging merge), used for Link-header dedup. */
+ final String requestUrl;
+
+ RestExchangeResult(
+ String body,
+ int status,
+ long responseTimeMs,
+ String headerJson,
+ MultivaluedMap<String, Object> headers,
+ String requestUrl) {
+ this.body = body;
+ this.status = status;
+ this.responseTimeMs = responseTimeMs;
+ this.headerJson = headerJson;
+ this.headers = headers;
+ this.requestUrl = requestUrl;
+ }
+ }
+
+ /** Mutable paging token state for {@link RestPaginationType} semantics. */
+ private static final class PaginationState {
+ int pageNumber = 1;
+ long offset = 0;
+
+ /** Next full URL supplied by RFC 5988 Link header paging. */
+ String linkNextUrl;
+
+ String cursorToken;
+ int effectiveLimit = 100;
+ }
+
public Rest(
TransformMeta transformMeta,
RestMeta meta,
@@ -100,17 +177,8 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
return queryParams;
}
- /**
- * Perform the rest call Ignore Sonar SSL warning, SSL can be disabled by a user action
- *
- * @param rowData
- * @return
- * @throws HopException
- */
- @SuppressWarnings("java:S5527")
- protected Object[] callRest(Object[] rowData) throws HopException {
-
- // get dynamic url ?
+ /** Resolves incoming-row URL substitution and HTTP method defaults for this row. */
+ protected void applyDynamicRowUrlAndMethod(Object[] rowData) throws HopException {
if (meta.isUrlInField()) {
if (!Utils.isEmpty(data.connectionName)) {
data.realUrl = baseUrl + data.inputRowMeta.getString(rowData, data.indexOfUrlField);
@@ -119,31 +187,74 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
}
}
- // get dynamic method?
if (meta.isDynamicMethod()) {
data.method = data.inputRowMeta.getString(rowData, data.indexOfMethod);
if (Utils.isEmpty(data.method)) {
throw new HopException(BaseMessages.getString(PKG, "Rest.Error.MethodMissing"));
}
}
+ }
+
+ /** Whether paging can run without degrading back to legacy single-call semantics. */
+ protected boolean supportsPaging() {
+ return meta.isPaginationEnabled()
+ && connection != null
+ && connection.getPaginationType() != null
+ && connection.getPaginationType() != RestPaginationType.NONE;
+ }
+
+ /**
+ * Merges additional query-string parameters onto an absolute REST URL intended for paging tokens.
+ */
+ protected String mergePagingQueriesIntoResolvedUrl(
+ String baseUrlResolved, LinkedHashMap<String, String> pagingQueries) {
+ if (pagingQueries == null || pagingQueries.isEmpty()) {
+ return baseUrlResolved;
+ }
+ UriBuilder ub = UriBuilder.fromUri(URI.create(baseUrlResolved));
+ for (Map.Entry<String, String> e : pagingQueries.entrySet()) {
+ if (!Utils.isEmpty(e.getKey())) {
+ ub.queryParam(e.getKey(), e.getValue() == null ? "" : e.getValue());
+ }
+ }
+ return ub.build().toString();
+ }
+
+ /**
+ * Executes a single REST exchange and bundles the deserialized pieces required for paging and row
+ * assembly.
+ */
+ @SuppressWarnings("java:S5527")
+ protected RestExchangeResult invokeRestExchange(
+ Object[] rowData,
+ String uriOverrideFull,
+ LinkedHashMap<String, String> pagingQueries,
+ LinkedHashMap<String, String> pagingBodyParams,
+ LinkedHashMap<String, String> pagingHeaderParams)
+ throws HopException {
+
+ applyDynamicRowUrlAndMethod(rowData);
+
+ String mergeBase = Utils.isEmpty(uriOverrideFull) ? data.realUrl : uriOverrideFull;
+ LinkedHashMap<String, String> query =
+ pagingQueries != null ? pagingQueries : new LinkedHashMap<>();
+ String effectiveBase = mergePagingQueriesIntoResolvedUrl(mergeBase, query);
+
WebTarget webResource = null;
Client client = null;
Invocation.Builder invocationBuilder = null;
- Object[] newRow = null;
- long startTime = 0;
- if (rowData != null) {
- newRow = rowData.clone();
- }
+ long startTime;
+ Response response;
final long httpLineageT0 = System.currentTimeMillis();
final long httpVolIn0 = dataVolumeIn != null ? dataVolumeIn : 0L;
final long httpVolOut0 = dataVolumeOut != null ? dataVolumeOut : 0L;
try {
if (isDetailed()) {
- logDetailed(BaseMessages.getString(PKG, "Rest.Log.ConnectingToURL", data.realUrl));
+ logDetailed(BaseMessages.getString(PKG, "Rest.Log.ConnectingToURL", effectiveBase));
}
if (!StringUtils.isEmpty(meta.getConnectionName())) {
invocationBuilder =
- connection.getInvocationBuilder(data.realUrl, data.realProxyHost, data.realProxyPort);
+ connection.getInvocationBuilder(effectiveBase, data.realProxyHost, data.realProxyPort);
} else {
ClientBuilder clientBuilder = createClientBuilder();
clientBuilder
@@ -159,14 +270,9 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
if (data.basicAuthentication != null) {
client.register(data.basicAuthentication);
}
- // create a WebResource object, which encapsulates a web resource for the client
- webResource = client.target(data.realUrl);
-
- // used for calculating the responseTime
- startTime = System.currentTimeMillis();
+ webResource = client.target(effectiveBase);
if (data.useMatrixParams) {
- // Add matrix parameters
UriBuilder builder = webResource.getUriBuilder();
for (int i = 0; i < data.nrMatrixParams; i++) {
String value = data.inputRowMeta.getString(rowData, data.indexOfMatrixParamFields[i]);
@@ -184,7 +290,6 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
}
if (data.useParams) {
- // Add query parameters
for (int i = 0; i < data.nrParams; i++) {
String value = data.inputRowMeta.getString(rowData, data.indexOfParamFields[i]);
if (isDebug()) {
@@ -204,20 +309,15 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
throw new HopException("Invocation builder not initialized");
}
- // set the Authentication/Authorization header from the connection first, if available.
- // this transform's headers will override this value if available.
MultivaluedMap<String, Object> headerMap = new MultivaluedHashMap<>();
addApiKeyHeaderIfAbsent(headerMap);
boolean acceptHeaderProvided = false;
- // media type override, if not null
String contentType = null;
if (data.useHeaders) {
- // Add headers
for (int i = 0; i < data.nrheader; i++) {
String value = data.inputRowMeta.getString(rowData, data.indexOfHeaderFields[i]);
- // unsure if an already set header will be returned to builder
headerMap.putSingle(data.headerNames[i], value);
if ("Content-Type".equals(data.headerNames[i])) {
contentType = value;
@@ -236,23 +336,46 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
headerMap.putSingle("Accept", data.mediaType);
}
- Response response = null;
+ /* Jersey Invocation.Builder.headers(MultivaluedMap) replaces all headers — so Bearer / API-key
+ * set inside RestConnection.getInvocationBuilder(...) would be stripped. Merge connection auth
+ * into outbound headers unless the caller already supplied Authorization. */
+ if (!StringUtils.isEmpty(meta.getConnectionName()) && connection != null) {
+ connection.applyBearerAndApiKeyHeaders(invocationBuilder, headerMap);
+ }
+
+ if (pagingHeaderParams != null && !pagingHeaderParams.isEmpty()) {
+ for (Map.Entry<String, String> e : pagingHeaderParams.entrySet()) {
+ if (!Utils.isEmpty(e.getKey())) {
+ headerMap.putSingle(e.getKey(), e.getValue() == null ? "" : e.getValue());
+ }
+ }
+ }
+
String entityString = null;
if (data.useBody) {
- // Set Http request entity
entityString = NVL(data.inputRowMeta.getString(rowData, data.indexOfBodyField), null);
if (isDebug()) {
logDebug(BaseMessages.getString(PKG, "Rest.Log.BodyValue", entityString));
}
}
+ if (pagingBodyParams != null && !pagingBodyParams.isEmpty()) {
+ String contentTypeForMerge = contentType;
+ if (Utils.isEmpty(contentTypeForMerge) && data.mediaType != null) {
+ contentTypeForMerge = data.mediaType.toString();
+ }
+ entityString =
+ PagingBodyMerge.merge(NVL(entityString, ""), pagingBodyParams, contentTypeForMerge);
+ if (isDebug()) {
+ logDebug(BaseMessages.getString(PKG, "Rest.Log.BodyValue", entityString));
+ }
+ }
- // execute first call
invocationBuilder.headers(headerMap);
final Invocation.Builder finalInvocationBuilder = invocationBuilder;
final String finalEntityString = entityString;
final String finalContentType = contentType;
- // add retry
+ startTime = System.currentTimeMillis();
response =
executeWithRetry(
() -> {
@@ -267,42 +390,26 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
if (response != null) {
response.bufferEntity();
}
- // Get response time
long responseTime = System.currentTimeMillis() - startTime;
- if (isDetailed()) {
- logDetailed(
- BaseMessages.getString(
- PKG, "Rest.Log.ResponseTime", String.valueOf(responseTime), data.realUrl));
- }
- // Get status
int status = response.getStatus();
- // Display status code
- if (isDebug()) {
- logDebug(BaseMessages.getString(PKG, "Rest.Log.ResponseCode", "" + status));
- }
- // Buffer the entity so we can read it multiple times if needed (for fallback)
if (response.hasEntity()) {
response.bufferEntity();
}
- // Get Response
String body;
- String headerString = null;
try {
body = response.readEntity(String.class);
} catch (ProcessingException ex) {
- // Check for duplicate Content-Type headers - this is a server configuration issue
String errorMessage = ex.getMessage();
if (errorMessage != null
&& errorMessage.contains("Too many \"Content-Type\" header values")) {
throw new HopException(
BaseMessages.getString(
- PKG, "Rest.Error.DuplicateContentType", data.realUrl, errorMessage),
+ PKG, "Rest.Error.DuplicateContentType", effectiveBase, errorMessage),
ex);
}
- // For other ProcessingExceptions, try fallback to raw InputStream
body = "";
if (response.hasEntity()) {
try (InputStream stream = response.readEntity(InputStream.class)) {
@@ -313,17 +420,11 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
if (isDetailed()) {
logDetailed("Unable to read response entity as String", ioEx);
}
- // If fallback also fails, throw with original exception context
throw new HopException(
- BaseMessages.getString(PKG, "Rest.Error.CanNotReadResponse", data.realUrl), ex);
- }
- } else {
- if (isDetailed()) {
- logDetailed("Response had no entity to read", ex);
+ BaseMessages.getString(PKG, "Rest.Error.CanNotReadResponse", effectiveBase), ex);
}
}
} catch (Exception ex) {
- // For non-ProcessingExceptions, try fallback to raw InputStream
body = "";
if (response.hasEntity()) {
try (InputStream stream = response.readEntity(InputStream.class)) {
@@ -335,19 +436,15 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
logDetailed("Unable to read response entity as String", ioEx);
}
throw new HopException(
- BaseMessages.getString(PKG, "Rest.Error.CanNotReadResponse", data.realUrl), ex);
- }
- } else {
- if (isDetailed()) {
- logDetailed("Response had no entity to read", ex);
+ BaseMessages.getString(PKG, "Rest.Error.CanNotReadResponse", effectiveBase), ex);
}
}
}
trackResponseBytes(response, body);
- // get Header
+
MultivaluedMap<String, Object> headers = searchForHeaders(response);
JSONObject json = new JSONObject();
- for (java.util.Map.Entry<String, List<Object>> entry : headers.entrySet()) {
+ for (Map.Entry<String, List<Object>> entry : headers.entrySet()) {
String name = entry.getKey();
List<Object> value = entry.getValue();
if (value.size() > 1) {
@@ -356,46 +453,645 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
json.put(name, value.get(0));
}
}
- headerString = json.toJSONString();
- // for output
- int returnFieldsOffset = data.inputRowMeta.size();
- // add response to output
- if (!Utils.isEmpty(data.resultFieldName)) {
- newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, body);
- returnFieldsOffset++;
- }
+ String headerString = json.toJSONString();
- // add status to output
- if (!Utils.isEmpty(data.resultCodeFieldName)) {
- newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, (long) status);
- returnFieldsOffset++;
- }
-
- // add response time to output
- if (!Utils.isEmpty(data.resultResponseFieldName)) {
- newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, responseTime);
- returnFieldsOffset++;
- }
- // add response header to output
- if (!Utils.isEmpty(data.resultHeaderFieldName)) {
- newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, headerString);
- }
emitHttpLineage(httpLineageT0, httpVolIn0, httpVolOut0, status, true, null);
+ return new RestExchangeResult(
+ body, status, responseTime, headerString, headers, effectiveBase);
} catch (Exception e) {
emitHttpLineage(httpLineageT0, httpVolIn0, httpVolOut0, null, false, e.getMessage());
throw new HopException(
- BaseMessages.getString(PKG, "Rest.Error.CanNotReadURL", data.realUrl), e);
+ BaseMessages.getString(PKG, "Rest.Error.CanNotReadURL", NVL(data.realUrl, effectiveBase)),
+ e);
} finally {
- if (webResource != null) {
- webResource = null;
- }
if (client != null) {
client.close();
}
}
+ }
+
+ protected Object[] assembleResultRow(Object[] baseRowMaybeNull, RestExchangeResult exchange)
+ throws HopException {
+
+ Object[] newRow = baseRowMaybeNull;
+ int returnFieldsOffset = data.inputRowMeta.size();
+ String body = exchange.body;
+ int status = exchange.status;
+ long responseTime = exchange.responseTimeMs;
+ String headerString = exchange.headerJson;
+
+ if (!Utils.isEmpty(data.resultFieldName)) {
+ newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, body);
+ returnFieldsOffset++;
+ }
+ if (!Utils.isEmpty(data.resultCodeFieldName)) {
+ newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, (long) status);
+ returnFieldsOffset++;
+ }
+ if (!Utils.isEmpty(data.resultResponseFieldName)) {
+ newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, responseTime);
+ returnFieldsOffset++;
+ }
+ if (!Utils.isEmpty(data.resultHeaderFieldName)) {
+ newRow = RowDataUtil.addValueData(newRow, returnFieldsOffset, headerString);
+ }
return newRow;
}
+ /**
+ * Legacy single-exchange flow (one output row built from exactly one REST response unless paging
+ * mode at {@link RestMeta#paginationEnabled}) is delegated through here by {@link #processRow()}
+ * when paging is inactive.
+ */
+ @SuppressWarnings("java:S5527")
+ protected Object[] callRest(Object[] rowData) throws HopException {
+ RestExchangeResult exchange =
+ invokeRestExchange(
+ rowData, null, new LinkedHashMap<>(), new LinkedHashMap<>(), new LinkedHashMap<>());
+ return assembleResultRow(rowData == null ? null : rowData.clone(), exchange);
+ }
+
+ private static String normalizeUrlForPagingDedup(String url) {
+ if (Utils.isEmpty(url)) {
+ return "";
+ }
+ try {
+ return URI.create(url.trim()).normalize().toASCIIString();
+ } catch (Exception e) {
+ return url.trim();
+ }
+ }
+
+ /** Follows paging instructions on the REST connection metadata for one incoming Hop row. */
+ protected void runPaginationLoop(Object[] rowData) throws HopException {
+ RestPaginationType pagingType =
+ connection != null ? connection.getPaginationType() : RestPaginationType.NONE;
+ PaginationState state = new PaginationState();
+ state.effectiveLimit = resolvePagingLimit(connection);
+
+ String resolvedSplitPath = resolveSplitPathOrPagingExpression(meta.getResultSplitPath());
+ int maxLoops =
+ meta.getMaxPagesLoops() > 0 ? meta.getMaxPagesLoops() : RestConst.DEFAULT_MAX_PAGES_LOOPS;
+
+ LinkedHashSet<String> linkPagingFetchedUrls = new LinkedHashSet<>();
+
+ applyDynamicRowUrlAndMethod(rowData);
+
+ boolean hasMore = true;
+
+ int loopIdx = 0;
+ int totalEmitted = 0;
+ RestExchangeResult firstExchange = null;
+ while (hasMore && loopIdx < maxLoops) {
+ String uriOv =
+ usesLinkStylePaging(pagingType) ? (loopIdx == 0 ? null : state.linkNextUrl) : null;
+
+ if (usesLinkStylePaging(pagingType) && uriOv != null) {
+ String nextKey = normalizeUrlForPagingDedup(uriOv);
+ if (linkPagingFetchedUrls.contains(nextKey)) {
+ logBasic(BaseMessages.getString(PKG, "Rest.Log.LinkPaginationStoppedRepeatedUrl", uriOv));
+ break;
+ }
+ }
+
+ LinkedHashMap<String, String> pageQs = buildPagingQuery(connection, pagingType, state);
+ LinkedHashMap<String, String> pageBody = buildPagingBody(connection, pagingType, state);
+ LinkedHashMap<String, String> pageHeaders = buildPagingHeaders(connection, pagingType, state);
+
+ RestExchangeResult ex = invokeRestExchange(rowData, uriOv, pageQs, pageBody, pageHeaders);
+ logBasic(
+ BaseMessages.getString(
+ PKG, "Rest.Log.PaginationFetchedPage", loopIdx + 1, NVL(ex.requestUrl, "")));
+ if (firstExchange == null) {
+ firstExchange = ex;
+ }
+ if (usesLinkStylePaging(pagingType)) {
+ String fetchedKey = normalizeUrlForPagingDedup(ex.requestUrl);
+ if (!linkPagingFetchedUrls.add(fetchedKey)) {
+ logBasic(
+ BaseMessages.getString(
+ PKG, "Rest.Log.LinkPaginationStoppedRepeatedUrl", NVL(ex.requestUrl, "")));
+ break;
+ }
+ }
+ totalEmitted += emitPagedResultRows(rowData, resolvedSplitPath, ex);
+
+ hasMore =
+ continuePaginationAfterExchange(connection, pagingType, state, resolvedSplitPath, ex);
+ loopIdx++;
+ if (!hasMore || loopIdx >= maxLoops && isDetailed()) {
+ if (loopIdx >= maxLoops && hasMore && isDetailed()) {
+ logDetailed(
+ "REST pagination stopped after safeguard maxPagesLoops=" + maxLoops + " iterations.");
+ }
+ }
+ }
+
+ enforcePagingSplitEmittedSomething(resolvedSplitPath, firstExchange, totalEmitted);
+ }
+
+ private static int resolvePagingLimit(RestConnection connection) {
+ if (connection == null) {
+ return 100;
+ }
+ RestPaginationType type = connection.getPaginationType();
+ if (RestPaginationType.OFFSET_LIMIT.equals(type)
+ || RestPaginationType.BODY_CURSOR.equals(type)
+ || RestPaginationType.HEADER_CURSOR.equals(type)) {
+ int lim = connection.getDefaultLimit();
+ return lim > 0 ? lim : 100;
+ }
+ return 100;
+ }
+
+ private LinkedHashMap<String, String> buildPagingQuery(
+ RestConnection conn, RestPaginationType type, PaginationState state) {
+ LinkedHashMap<String, String> q = new LinkedHashMap<>();
+ if (conn == null || type == null || RestPaginationType.NONE.equals(type)) {
+ return q;
+ }
+ switch (type) {
+ case OFFSET_LIMIT -> {
+ String offKey = NVL(resolve(conn.getOffsetParamName()), "offset");
+ String limKey = NVL(resolve(conn.getLimitParamName()), "limit");
+ q.put(offKey, Long.toString(state.offset));
+ q.put(limKey, Integer.toString(state.effectiveLimit));
+ }
+ case PAGE_NUMBER ->
+ q.put(NVL(resolve(conn.getPageParamName()), "page"), Integer.toString(state.pageNumber));
+ case CURSOR -> {
+ if (!Utils.isEmpty(state.cursorToken)) {
+ q.put(NVL(resolve(conn.getPageParamName()), "cursor"), state.cursorToken);
+ }
+ }
+ case BODY_CURSOR -> appendCursorBatchToQueryIfGet(conn, state, q);
+ case HEADER_CURSOR, LINK_HEADER, BODY_NEXT_URL, NONE -> {
+ /* header, URL, or body carries paging tokens */
+ }
+ default -> {
+ /* exhaustive */
+ }
+ }
+ return q;
+ }
+
+ private LinkedHashMap<String, String> buildPagingBody(
+ RestConnection conn, RestPaginationType type, PaginationState state) {
+ LinkedHashMap<String, String> body = new LinkedHashMap<>();
+ if (conn == null || type == null || !RestPaginationType.BODY_CURSOR.equals(type)) {
+ return body;
+ }
+ if (isGetPagingMethod()) {
+ return body;
+ }
+ appendCursorBatchParams(conn, state, body);
+ return body;
+ }
+
+ private LinkedHashMap<String, String> buildPagingHeaders(
+ RestConnection conn, RestPaginationType type, PaginationState state) {
+ LinkedHashMap<String, String> headers = new LinkedHashMap<>();
+ if (conn == null || type == null || !RestPaginationType.HEADER_CURSOR.equals(type)) {
+ return headers;
+ }
+ appendCursorBatchParams(conn, state, headers);
+ return headers;
+ }
+
+ private void appendCursorBatchToQueryIfGet(
+ RestConnection conn, PaginationState state, LinkedHashMap<String, String> query) {
+ if (!isGetPagingMethod()) {
+ return;
+ }
+ appendCursorBatchParams(conn, state, query);
+ }
+
+ private void appendCursorBatchParams(
+ RestConnection conn, PaginationState state, LinkedHashMap<String, String> target) {
+ if (state.effectiveLimit > 0) {
+ target.put(
+ NVL(resolve(conn.getLimitParamName()), "limit"), Integer.toString(state.effectiveLimit));
+ }
+ if (!Utils.isEmpty(state.cursorToken)) {
+ target.put(NVL(resolve(conn.getPageParamName()), "cursor"), state.cursorToken);
+ }
+ }
+
+ private boolean isGetPagingMethod() {
+ return RestMeta.HTTP_METHOD_GET.equals(NVL(data.method, ""));
+ }
+
+ private static boolean usesLinkStylePaging(RestPaginationType pagingType) {
+ return RestPaginationType.LINK_HEADER.equals(pagingType)
+ || RestPaginationType.BODY_NEXT_URL.equals(pagingType);
+ }
+
+ private boolean continuePaginationAfterExchange(
+ RestConnection conn,
+ RestPaginationType pagingType,
+ PaginationState state,
+ String resolvedSplitPath,
+ RestExchangeResult ex)
+ throws HopException {
+
+ if (conn == null || pagingType == null || RestPaginationType.NONE.equals(pagingType)) {
+ return false;
+ }
+
+ boolean httpOk = ex.status >= 200 && ex.status < 300;
+ boolean hadPayload = exchangeHadPayload(ex, resolvedSplitPath);
+
+ if (!httpOk) {
+ return false;
+ }
+
+ switch (pagingType) {
+ case LINK_HEADER -> {
+ state.linkNextUrl = extractRelNextUri(ex.headers);
+ return !Utils.isEmpty(state.linkNextUrl);
+ }
+ case BODY_NEXT_URL -> {
+ state.linkNextUrl = extractNextPageUrlFromBody(conn, ex.body).orElse(null);
+ return !Utils.isEmpty(state.linkNextUrl);
+ }
+ case OFFSET_LIMIT -> {
+ if (!hadPayload) {
+ return false;
+ }
+ state.offset += state.effectiveLimit;
+ return true;
+ }
+ case PAGE_NUMBER -> {
+ if (!hadPayload) {
+ return false;
+ }
+ state.pageNumber++;
+ return true;
+ }
+ case CURSOR -> {
+ if (!hadPayload) {
+ return false;
+ }
+ return extractCursorForNext(conn, ex.body)
+ .map(
+ tok -> {
+ state.cursorToken = tok;
+ return true;
+ })
+ .orElse(false);
+ }
+ case BODY_CURSOR, HEADER_CURSOR -> {
+ return extractCursorForNext(conn, ex.body)
+ .map(
+ tok -> {
+ state.cursorToken = tok;
+ return true;
+ })
+ .orElse(false);
+ }
+ case NONE -> {
+ return false;
+ }
+ default -> {
+ return false;
+ }
+ }
+ }
+
+ /**
+ * @return whether this response appears to carry at least one item (whole body when no
+ * split-path, or extracted elements when splitting).
+ */
+ private boolean exchangeHadPayload(RestExchangeResult ex, String resolvedSplitPath)
+ throws HopException {
+ List<String> items =
+ Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, resolvedSplitPath);
+ if (items != null) {
+ return !items.isEmpty();
+ }
+ return !Utils.isEmpty(NVL(ex.body, "").trim());
+ }
+
+ /** Uses RFC 5988 / RFC 8288 Web Link parsing ({@code LinkHeaderPaging}) for {@code rel=next}. */
+ private static String extractRelNextUri(MultivaluedMap<String, Object> headers) {
+ return LinkHeaderPaging.findFirstUriWithRelNext(headers);
+ }
+
+ /**
+ * Resolves %%VAR%% and ${VAR} for JsonPath / XPath paging expressions without applying Hop's hex
+ * notation ($[hh],…) from {@link StringUtil#environmentSubstitute}, which would corrupt common
+ * JsonPath literals such as "$[*]" (treated as a malformed hex escape).
+ */
+ private String resolveSplitPathOrPagingExpression(String literal) {
+ if (literal == null) {
+ return null;
+ }
+ if (Utils.isEmpty(literal)) {
+ return literal;
+ }
+ Map<String, String> map = substitutionMapSnapshot();
+ String afterWindows = StringUtil.substituteWindows(literal, map);
+ return StringUtil.substituteUnix(afterWindows, map);
+ }
+
+ private HashMap<String, String> substitutionMapSnapshot() {
+ HashMap<String, String> map = new HashMap<>();
+ for (String name : getVariableNames()) {
+ String v = getVariable(name);
+ if (v != null) {
+ map.put(name, v);
+ }
+ }
+ return map;
+ }
+
+ /**
+ * Paging plus result split emits nothing for many failure modes (wrong JsonPath, error JSON body,
+ * auth failures returning objects, etc.). Failing loudly avoids silent pipelines with zero
+ * output.
+ *
+ * <p>Legitimate empty-collection cases (no rows, no error): HTTP 2xx with a top-level {@code []},
+ * or JSON where the split path points at an existing but empty array (e.g. Shopify {@code
+ * {"orders":[]}} with split {@code $.orders[*]}).
+ */
+ private void enforcePagingSplitEmittedSomething(
+ String resolvedSplitPath, RestExchangeResult firstExchange, int totalEmitted)
+ throws HopException {
+ if (!meta.isPaginationEnabled() || Utils.isEmpty(resolvedSplitPath)) {
+ return;
+ }
+ if (totalEmitted > 0) {
+ return;
+ }
+ if (firstExchange == null) {
+ throw new HopException(
+ BaseMessages.getString(PKG, "Rest.Error.PagingSplitProducedZeroRowsNoResponse"));
+ }
+ boolean httpOk = firstExchange.status >= 200 && firstExchange.status < 300;
+ String bodyTrim = NVL(firstExchange.body, "").trim();
+ if (firstExchange.status == 401 || firstExchange.status == 403) {
+ throw new HopException(
+ BaseMessages.getString(
+ PKG,
+ "Rest.Error.PagingAuthFailed",
+ Integer.toString(firstExchange.status),
+ abbreviateBodyForLog(firstExchange.body, 900)));
+ }
+ if (httpOk && isLegitimateEmptyPagedSplit(bodyTrim, resolvedSplitPath)) {
+ logBasic(
+ BaseMessages.getString(
+ PKG, "Rest.Log.PagingEmptyCollectionFirstPage", NVL(resolvedSplitPath, "")));
+ return;
+ }
+
+ throw new HopException(
+ BaseMessages.getString(
+ PKG,
+ "Rest.Error.PagingSplitProducedZeroRows",
+ Integer.toString(firstExchange.status),
+ NVL(resolvedSplitPath, ""),
+ abbreviateBodyForLog(firstExchange.body, 900)));
+ }
+
+ /**
+ * True when HTTP succeeded but the split path targets an empty collection (not a missing/wrong
+ * path). Supports top-level {@code []} and wrapped arrays such as {@code {"orders":[]}}.
+ */
+ boolean isLegitimateEmptyPagedSplit(String bodyTrim, String resolvedSplitPath) {
+ if (Utils.isEmpty(bodyTrim) || Utils.isEmpty(resolvedSplitPath)) {
+ return false;
+ }
+ if ("[]".equals(bodyTrim)) {
+ return true;
+ }
+ if (!RestMeta.APPLICATION_TYPE_JSON.equals(NVL(resolve(meta.getApplicationType()), ""))) {
+ return false;
+ }
+ try {
+ String parentPath = resolvedSplitPath.replaceAll("\\[\\*\\]\\s*$", "");
+ String jsonPathToProbe =
+ parentPath.equals(resolvedSplitPath) ? resolvedSplitPath : parentPath;
+ Object atPath = JsonPath.using(JSON_PATH_CONFIGURATION).parse(bodyTrim).read(jsonPathToProbe);
+ if (atPath == null) {
+ return false;
+ }
+ if (atPath instanceof Iterable<?> iterable) {
+ return !iterable.iterator().hasNext();
+ }
+ // Single object at path without [*] — zero rows means nothing to split, still valid.
+ return true;
+ } catch (Exception ignored) {
+ return false;
+ }
+ }
+
+ private int emitPagedResultRows(
+ Object[] rowTemplate, String resolvedSplitPath, RestExchangeResult ex) throws HopException {
+
+ String splitPathResolved = Utils.isEmpty(resolvedSplitPath) ? null : resolvedSplitPath;
+ if (splitPathResolved == null) {
+ putRow(data.outputRowMeta, assembleResultRow(rowTemplate.clone(), ex));
+ return 1;
+ }
+
+ List<String> chunks = splitResultItems(ex.body, splitPathResolved);
+ if (chunks.isEmpty()) {
+ return 0;
+ }
+
+ int emitted = 0;
+ for (String sliceBody : chunks) {
+ RestExchangeResult slice =
+ new RestExchangeResult(
+ sliceBody, ex.status, ex.responseTimeMs, ex.headerJson, ex.headers, ex.requestUrl);
+ putRow(data.outputRowMeta, assembleResultRow(rowTemplate.clone(), slice));
+ emitted++;
+ }
+ return emitted;
+ }
+
+ private List<String> splitResultItems(String body, String jsonOrXPathExpr) throws HopException {
+ String appType = NVL(resolve(meta.getApplicationType()), "");
+
+ List<String> out = new ArrayList<>();
+ try {
+ if (RestMeta.APPLICATION_TYPE_JSON.equals(appType)) {
+ Object raw =
+ JsonPath.using(JSON_PATH_CONFIGURATION).parse(NVL(body, "{}")).read(jsonOrXPathExpr);
+ appendStructuredJsonPieces(out, raw);
+ return out;
+ }
+ if (RestMeta.APPLICATION_TYPE_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_SVG_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_XHTML.equals(appType)) {
+
+ DocumentBuilderFactory df = documentBuilderFactory();
+ DocumentBuilder db = df.newDocumentBuilder();
+ Document doc;
+ try (StringReader sr = new StringReader(NVL(body, ""))) {
+ doc = db.parse(new InputSource(sr));
+ }
+ XPathFactory xpf = XPathFactory.newInstance();
+ XPath xp = xpf.newXPath();
+ XPathExpression expr = xp.compile(jsonOrXPathExpr);
+ NodeList nl = (NodeList) expr.evaluate(doc, XPathConstants.NODESET);
+
+ TransformerFactory tf = TransformerFactory.newInstance();
+ try {
+ tf.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
+ } catch (Exception ignored) {
+ // Older JDKs ignore unknown features gracefully.
+ }
+ Transformer transformer = tf.newTransformer();
+ transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
+
+ for (int i = 0; i < nl.getLength(); i++) {
+ StringWriter sw = new StringWriter();
+ transformer.transform(new DOMSource(nl.item(i)), new StreamResult(sw));
+ String piece = sw.toString();
+ if (!Utils.isEmpty(piece.trim())) {
+ out.add(piece);
+ }
+ }
+ return out;
+ }
+
+ throw new HopException(
+ "REST resultSplitPath requires application type XML or JSON, got '" + appType + "'.");
+
+ } catch (HopException he) {
+ throw he;
+ } catch (Exception e) {
+ throw new HopException("Unable to split REST response payload", e);
+ }
+ }
+
+ private DocumentBuilderFactory documentBuilderFactory() throws Exception {
+ DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
+ dbf.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
+ dbf.setNamespaceAware(true);
+ return dbf;
+ }
+
+ private void appendStructuredJsonPieces(List<String> out, Object raw) throws Exception {
+ if (raw == null) {
+ return;
+ }
+ if (raw instanceof Iterable<?> it) {
+ for (Object elem : it) {
+ stringifyJsonPiece(out, elem);
+ }
+ return;
+ }
+ stringifyJsonPiece(out, raw);
+ }
+
+ private void stringifyJsonPiece(List<String> out, Object o) throws Exception {
+ if (o == null) {
+ return;
+ }
+ try {
+ out.add(paginationJsonMapper.writeValueAsString(o));
+ } catch (Exception e) {
+ out.add(o.toString());
+ }
+ }
+
+ private java.util.Optional<String> extractCursorForNext(RestConnection conn, String body) {
+ if (conn == null) {
+ return java.util.Optional.empty();
+ }
+ String appType = NVL(resolve(meta.getApplicationType()), "");
+ try {
+ if (RestMeta.APPLICATION_TYPE_JSON.equals(appType)) {
+ if (Utils.isEmpty(conn.getCursorJsonPath())) {
+ return java.util.Optional.empty();
+ }
+ Object raw =
+ JsonPath.using(JSON_PATH_CONFIGURATION)
+ .parse(NVL(body, "{}"))
+ .read(resolve(conn.getCursorJsonPath()));
+ if (raw == null) {
+ return java.util.Optional.empty();
+ }
+ String s = raw.toString().trim();
+ return s.isEmpty() ? java.util.Optional.empty() : java.util.Optional.of(s);
+ }
+
+ if (RestMeta.APPLICATION_TYPE_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)) {
+
+ if (Utils.isEmpty(conn.getCursorXPath())) {
+ return java.util.Optional.empty();
+ }
+ DocumentBuilderFactory df = documentBuilderFactory();
+ Document doc;
+ try (StringReader sr = new StringReader(NVL(body, ""))) {
+ doc = df.newDocumentBuilder().parse(new InputSource(sr));
+ }
+ XPathFactory xpf = XPathFactory.newInstance();
+ XPathExpression expr =
+ xpf.newXPath().compile(resolveSplitPathOrPagingExpression(conn.getCursorXPath()));
+ String val = expr.evaluate(doc);
+ String s = val == null ? "" : val.trim();
+ return s.isEmpty() ? java.util.Optional.empty() : java.util.Optional.of(s);
+ }
+ } catch (Exception ignored) {
+ /* JsonPath suppressed option may yield null upstream */
+ }
+ return java.util.Optional.empty();
+ }
+
+ private java.util.Optional<String> extractNextPageUrlFromBody(RestConnection conn, String body) {
+ if (conn == null) {
+ return java.util.Optional.empty();
+ }
+ String appType = NVL(resolve(meta.getApplicationType()), "");
+ try {
+ if (RestMeta.APPLICATION_TYPE_JSON.equals(appType)) {
+ if (Utils.isEmpty(conn.getNextPageUrlJsonPath())) {
+ return java.util.Optional.empty();
+ }
+ Object raw =
+ JsonPath.using(JSON_PATH_CONFIGURATION)
+ .parse(NVL(body, "{}"))
+ .read(resolveSplitPathOrPagingExpression(conn.getNextPageUrlJsonPath()));
+ return normalizeNextPageUrl(raw);
+ }
+
+ if (RestMeta.APPLICATION_TYPE_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)) {
+ if (Utils.isEmpty(conn.getNextPageUrlXPath())) {
+ return java.util.Optional.empty();
+ }
+ DocumentBuilderFactory df = documentBuilderFactory();
+ Document doc;
+ try (StringReader sr = new StringReader(NVL(body, ""))) {
+ doc = df.newDocumentBuilder().parse(new InputSource(sr));
+ }
+ XPathFactory xpf = XPathFactory.newInstance();
+ XPathExpression expr =
+ xpf.newXPath().compile(resolveSplitPathOrPagingExpression(conn.getNextPageUrlXPath()));
+ return normalizeNextPageUrl(expr.evaluate(doc));
+ }
+ } catch (Exception ignored) {
+ /* suppressed JsonPath / XPath failures */
+ }
+ return java.util.Optional.empty();
+ }
+
+ private static java.util.Optional<String> normalizeNextPageUrl(Object raw) {
+ if (raw == null) {
+ return java.util.Optional.empty();
+ }
+ String s = raw.toString().trim();
+ return s.isEmpty() ? java.util.Optional.empty() : java.util.Optional.of(s);
+ }
+
private Response executeWithRetry(Supplier<Response> requestSupplier) throws HopException {
int maxRetries =
meta.getRetryTimes() != null ? meta.getRetryTimes() : RestConst.DEFAULT_RETRY_TIMES;
@@ -632,6 +1328,14 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
return StandardCharsets.UTF_8;
}
+ private static String abbreviateBodyForLog(String body, int maxChars) {
+ String s = NVL(body, "").replaceAll("\\s+", " ").trim();
+ if (s.length() <= maxChars) {
+ return s.isEmpty() ? "(empty)" : s;
+ }
+ return s.substring(0, maxChars) + "...";
+ }
+
/**
* set the Authentication/Authorization header from the connection first, if available. this
* transform's headers will override this value if available.
@@ -643,7 +1347,7 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
return;
}
- if (!"API Key".equals(connection.getAuthType())) {
+ if (!"API Key".equalsIgnoreCase(connection.getAuthType())) {
return;
}
@@ -873,8 +1577,17 @@ public class Rest extends BaseTransform<RestMeta, RestData> {
}
} // end if first
try {
- Object[] outputRowData = callRest(r);
- putRow(data.outputRowMeta, outputRowData); // copy row to output rowset(s)
+ if (supportsPaging()) {
+ runPaginationLoop(r);
+ } else {
+ if (meta.isPaginationEnabled() && isDetailed()) {
+ logDetailed(
+ "REST pagination is configured on this transform but is inactive "
+ + "(needs a REST connection with a non-NONE pagination type). Using legacy single-request behaviour.");
+ }
+ Object[] outputRowData = callRest(r);
+ putRow(data.outputRowMeta, outputRowData);
+ }
if (checkFeedback(getLinesRead()) && isDetailed()) {
logDetailed(BaseMessages.getString(PKG, "Rest.LineNumber") + getLinesRead());
}
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestDialog.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestDialog.java
index 5ba5c90c0c..c57d8d3d2c 100644
--- a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestDialog.java
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestDialog.java
@@ -159,6 +159,20 @@ public class RestDialog extends BaseTransformDialog {
private TableView wStatusCodes;
private TableView wMethods;
+ private Label wlPaginationInfo;
+
+ private Label wlPaginationEnabled;
+
+ private Button wPaginationEnabled;
+
+ private Label wlMaxPagesLoops;
+
+ private TextVar wMaxPagesLoops;
+
+ private Label wlSplitPath;
+
+ private TextVar wResultSplitPath;
+
public RestDialog(
Shell parent, IVariables variables, RestMeta transformMeta, PipelineMeta pipelineMeta) {
super(parent, variables, transformMeta, pipelineMeta);
@@ -419,7 +433,8 @@ public class RestDialog extends BaseTransformDialog {
setupMatrixParamTabContent(lsMod, margin, wMatrixParametersTab, wMatrixParametersComp);
- // retry
+ addPaginationTabItem(wTabFolder, lsMod);
+
addRetryTabItem(wTabFolder, lsMod);
FormData fdTabFolder = new FormData();
@@ -444,6 +459,7 @@ public class RestDialog extends BaseTransformDialog {
activateUrlInfield();
activateMethodInfield();
activateTrustStoreFields();
+ activatePaginationTab();
setMethod();
input.setChanged(changed);
focusTransformName();
@@ -452,6 +468,97 @@ public class RestDialog extends BaseTransformDialog {
return transformName;
}
+ private void addPaginationTabItem(CTabFolder wTabFolder, ModifyListener lsMod) {
+ CTabItem paginationTab = new CTabItem(wTabFolder, SWT.NONE);
+ paginationTab.setFont(GuiResource.getInstance().getFontDefault());
+ paginationTab.setText(BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.Title"));
+
+ Composite wPagComp = new Composite(wTabFolder, SWT.NONE);
+ PropsUi.setLook(wPagComp);
+ FormLayout pl = new FormLayout();
+ pl.marginWidth = PropsUi.getFormMargin();
+ pl.marginHeight = PropsUi.getFormMargin();
+ wPagComp.setLayout(pl);
+
+ wlPaginationInfo = new Label(wPagComp, SWT.LEFT | SWT.WRAP);
+ wlPaginationInfo.setText(BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.Info"));
+ PropsUi.setLook(wlPaginationInfo);
+ FormData fdInfo = new FormData();
+ fdInfo.left = new FormAttachment(0, margin);
+ fdInfo.right = new FormAttachment(100, -margin);
+ fdInfo.top = new FormAttachment(0, margin);
+ wlPaginationInfo.setLayoutData(fdInfo);
+
+ wlPaginationEnabled = new Label(wPagComp, SWT.RIGHT);
+ wlPaginationEnabled.setText(
+ BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.Enable.Label"));
+ PropsUi.setLook(wlPaginationEnabled);
+ wlPaginationEnabled.setToolTipText(
+ BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.Enable.Tooltip"));
+ FormData fdlEn = new FormData();
+ fdlEn.left = new FormAttachment(0, margin);
+ fdlEn.right = new FormAttachment(middle, -margin);
+ fdlEn.top = new FormAttachment(wlPaginationInfo, margin);
+ wlPaginationEnabled.setLayoutData(fdlEn);
+
+ wPaginationEnabled = new Button(wPagComp, SWT.CHECK);
+ PropsUi.setLook(wPaginationEnabled);
+ wPaginationEnabled.setToolTipText(wlPaginationEnabled.getToolTipText());
+ wPaginationEnabled.addListener(SWT.Selection, e -> input.setChanged());
+ FormData fdEn = new FormData();
+ fdEn.left = new FormAttachment(middle, margin);
+ fdEn.right = new FormAttachment(100, -margin);
+ fdEn.top = new FormAttachment(wlPaginationEnabled, 0, SWT.CENTER);
+ wPaginationEnabled.setLayoutData(fdEn);
+
+ wlMaxPagesLoops = new Label(wPagComp, SWT.RIGHT);
+ wlMaxPagesLoops.setText(
+ BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.MaxPagesLoops.Label"));
+ PropsUi.setLook(wlMaxPagesLoops);
+ wlMaxPagesLoops.setToolTipText(
+ BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.MaxPagesLoops.Tooltip"));
+
+ FormData fdlMax = new FormData();
+ fdlMax.left = new FormAttachment(0, margin);
+ fdlMax.right = new FormAttachment(middle, -margin);
+ fdlMax.top = new FormAttachment(wPaginationEnabled, margin);
+ wlMaxPagesLoops.setLayoutData(fdlMax);
+
+ wMaxPagesLoops = new TextVar(variables, wPagComp, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wMaxPagesLoops);
+ wMaxPagesLoops.setToolTipText(wlMaxPagesLoops.getToolTipText());
+ wMaxPagesLoops.addModifyListener(lsMod);
+
+ FormData fdMax = new FormData();
+ fdMax.left = new FormAttachment(middle, margin);
+ fdMax.right = new FormAttachment(100, -margin);
+ fdMax.top = new FormAttachment(wlMaxPagesLoops, 0, SWT.CENTER);
+ wMaxPagesLoops.setLayoutData(fdMax);
+
+ wlSplitPath = new Label(wPagComp, SWT.RIGHT);
+ wlSplitPath.setText(BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.SplitPath.Label"));
+ PropsUi.setLook(wlSplitPath);
+ wlSplitPath.setToolTipText(
+ BaseMessages.getString(PKG, "RestDialog.Tab.Pagination.SplitPath.Tooltip"));
+ FormData fdlSplit = new FormData();
+ fdlSplit.left = new FormAttachment(0, margin);
+ fdlSplit.right = new FormAttachment(middle, -margin);
+ fdlSplit.top = new FormAttachment(wMaxPagesLoops, margin);
+ wlSplitPath.setLayoutData(fdlSplit);
+
+ wResultSplitPath = new TextVar(variables, wPagComp, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wResultSplitPath);
+ wResultSplitPath.setToolTipText(wlSplitPath.getToolTipText());
+ wResultSplitPath.addModifyListener(lsMod);
+ FormData fdSplitTxt = new FormData();
+ fdSplitTxt.left = new FormAttachment(middle, margin);
+ fdSplitTxt.right = new FormAttachment(100, -margin);
+ fdSplitTxt.top = new FormAttachment(wlSplitPath, 0, SWT.CENTER);
+ wResultSplitPath.setLayoutData(fdSplitTxt);
+
+ paginationTab.setControl(wPagComp);
+ }
+
private void addRetryTabItem(CTabFolder wTabFolder, ModifyListener lsMod) {
CTabItem retryTab = new CTabItem(wTabFolder, SWT.NONE);
retryTab.setFont(GuiResource.getInstance().getFontDefault());
@@ -768,6 +875,28 @@ public class RestDialog extends BaseTransformDialog {
});
}
+ private void activatePaginationTab() {
+ boolean hasConnection = wSelectionLine != null && !Utils.isEmpty(wSelectionLine.getText());
+ if (wlPaginationEnabled != null && !wlPaginationEnabled.isDisposed()) {
+ wlPaginationEnabled.setEnabled(hasConnection);
+ }
+ if (wPaginationEnabled != null && !wPaginationEnabled.isDisposed()) {
+ wPaginationEnabled.setEnabled(hasConnection);
+ }
+ if (wlMaxPagesLoops != null && !wlMaxPagesLoops.isDisposed()) {
+ wlMaxPagesLoops.setEnabled(hasConnection);
+ }
+ if (wMaxPagesLoops != null && !wMaxPagesLoops.isDisposed()) {
+ wMaxPagesLoops.setEnabled(hasConnection);
+ }
+ if (wlSplitPath != null && !wlSplitPath.isDisposed()) {
+ wlSplitPath.setEnabled(hasConnection);
+ }
+ if (wResultSplitPath != null && !wResultSplitPath.isDisposed()) {
+ wResultSplitPath.setEnabled(hasConnection);
+ }
+ }
+
private void activateTrustStoreFields() {
// trust store file(label/text/browse)
wlTrustStoreFile.setEnabled(!wIgnoreSsl.getSelection());
@@ -1330,7 +1459,17 @@ public class RestDialog extends BaseTransformDialog {
fdSelectionLine.top = new FormAttachment(wGeneralComp, margin);
fdSelectionLine.right = new FormAttachment(100, -margin);
wSelectionLine.setLayoutData(fdSelectionLine);
- wSelectionLine.addListener(SWT.Selection, e -> input.setChanged(true));
+ wSelectionLine.addListener(
+ SWT.Selection,
+ e -> {
+ input.setChanged(true);
+ activatePaginationTab();
+ });
+ wSelectionLine.addModifyListener(
+ e -> {
+ input.setChanged(true);
+ activatePaginationTab();
+ });
try {
wSelectionLine.fillItems();
} catch (Exception e) {
@@ -1540,6 +1679,18 @@ public class RestDialog extends BaseTransformDialog {
// get retry data
getDataForRetry();
+
+ getDataForPagination();
+ }
+
+ private void getDataForPagination() {
+ wPaginationEnabled.setSelection(input.isPaginationEnabled());
+ if (input.getMaxPagesLoops() > 0) {
+ wMaxPagesLoops.setText(Integer.toString(input.getMaxPagesLoops()));
+ } else {
+ wMaxPagesLoops.setText("");
+ }
+ wResultSplitPath.setText(Const.NVL(input.getResultSplitPath(), ""));
}
private void getDataForRetry() {
@@ -1649,10 +1800,26 @@ public class RestDialog extends BaseTransformDialog {
input.setApplicationType(wApplicationType.getText());
transformName = wTransformName.getText(); // return value
+ okForPagination();
okForRetry();
dispose();
}
+ private void okForPagination() {
+ input.setPaginationEnabled(wPaginationEnabled.getSelection());
+ if (!Utils.isEmpty(wMaxPagesLoops.getText())) {
+ try {
+ int parsed = Integer.parseInt(variables.resolve(Const.trim(wMaxPagesLoops.getText())));
+ input.setMaxPagesLoops(Math.max(parsed, 0));
+ } catch (NumberFormatException ex) {
+ input.setMaxPagesLoops(0);
+ }
+ } else {
+ input.setMaxPagesLoops(0);
+ }
+ input.setResultSplitPath(wResultSplitPath.getText());
+ }
+
private void okForRetry() {
// retry times
if (!Utils.isEmpty(wRetryTimes.getText())) {
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestMeta.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestMeta.java
index 3fbe597c1b..a0fee99c21 100644
--- a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestMeta.java
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/RestMeta.java
@@ -219,6 +219,20 @@ public class RestMeta extends BaseTransformMeta<Rest, RestData> {
injectionGroupKey = "RETRY_METHODS")
private List<String> retryMethods;
+ @HopMetadataProperty(key = "paginationEnabled", injectionKey = "PAGINATION_ENABLED")
+ private boolean paginationEnabled;
+
+ @HopMetadataProperty(key = "maxPagesLoops", injectionKey = "MAX_PAGES_LOOPS")
+ private int maxPagesLoops;
+
+ /**
+ * Optional JsonPath ({@link #APPLICATION_TYPE_JSON}) or XPath ({@link #APPLICATION_TYPE_XML})
+ * that selects an array or node-set; each matched element becomes one outgoing row instead of
+ * buffering whole responses.
+ */
+ @HopMetadataProperty(key = "resultSplitPath", injectionKey = "RESULT_SPLIT_PATH")
+ private String resultSplitPath;
+
public RestMeta() {
super(); // allocate BaseTransformMeta
headerFields = new ArrayList<>();
@@ -259,6 +273,10 @@ public class RestMeta extends BaseTransformMeta<Rest, RestData> {
this.retryDelayMs = RestConst.DEFAULT_RETRY_DELAY_MS;
this.retryStatusCodes.addAll(RestConst.retryStatusCodes());
this.retryMethods.addAll(RestConst.retryMethods());
+
+ this.paginationEnabled = false;
+ this.maxPagesLoops = RestConst.DEFAULT_MAX_PAGES_LOOPS;
+ this.resultSplitPath = null;
}
@Override
diff --git a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/common/RestConst.java b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/common/RestConst.java
index 8b2e346e41..2074da276d 100644
--- a/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/common/RestConst.java
+++ b/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/common/RestConst.java
@@ -27,6 +27,9 @@ public class RestConst {
public static final int DEFAULT_RETRY_TIMES = 0;
public static final long DEFAULT_RETRY_DELAY_MS = 200L;
+ /** Maximum pages fetched per incoming row when pagination is enabled (safeguard). */
+ public static final int DEFAULT_MAX_PAGES_LOOPS = 256;
+
/**
* By default, the HTTP status codes that should be retried include 429, 500, 502, 503, and 504.
*/
diff --git a/plugins/transforms/rest/src/main/resources/org/apache/hop/pipeline/transforms/rest/messages/messages_en_US.properties b/plugins/transforms/rest/src/main/resources/org/apache/hop/pipeline/transforms/rest/messages/messages_en_US.properties
index 545f466c85..ab62fe61d2 100644
--- a/plugins/transforms/rest/src/main/resources/org/apache/hop/pipeline/transforms/rest/messages/messages_en_US.properties
+++ b/plugins/transforms/rest/src/main/resources/org/apache/hop/pipeline/transforms/rest/messages/messages_en_US.properties
@@ -43,6 +43,13 @@ Rest.Log.NoField=URL field name is missing\!
Rest.Log.queryParameterValue=Query parameter [{0}]\=''{1}''
Rest.Log.ResponseCode=The response code is {0}
Rest.Log.ResponseTime=Response time (milliseconds)\: [{0}] for [{1}]
+Rest.Log.PagingEmptyJsonArrayFirstPage=REST paging: HTTP success with an empty JSON array ([]); no rows to emit.
+Rest.Log.PagingEmptyCollectionFirstPage=REST paging: HTTP success with an empty collection at split path "{0}"; no rows to emit.
+Rest.Log.LinkPaginationStoppedRepeatedUrl=REST Link header pagination stopped: the next request URL was already fetched ({0}), avoiding an infinite loop when rel=\"next\" repeats or cycles.
+Rest.Log.PaginationFetchedPage=REST pagination: fetched results page {0} from [{1}]
+Rest.Error.PagingSplitProducedZeroRows=REST paging with a result split path produced no rows. Last HTTP status: {0}. Split path used: "{1}". First response excerpt: {2}
+Rest.Error.PagingAuthFailed=REST paging stopped: HTTP {0} (authentication or permission denied). Check the REST connection (Bearer token, API key header, and resolved project/environment variables). Response excerpt: {1}
+Rest.Error.PagingSplitProducedZeroRowsNoResponse=REST paging with a result split path produced no rows and no HTTP response was recorded (unexpected).
Rest.Name=REST client
RestDialog.ApplicationType.Label=Application type
RestDialog.Auth.Title=Authentication
@@ -117,4 +124,12 @@ RestDialog.Tab.Retry.Delay=Retry delay(ms)
RestDialog.Tab.Retry.Http.Status=Retry HTTP status codes
RestDialog.Tab.Retry.Http.Status.Header=Status Code
RestDialog.Tab.Retry.Http.Method=Retry HTTP methods
-RestDialog.Tab.Retry.Http.Method.Header=Method
\ No newline at end of file
+RestDialog.Tab.Retry.Http.Method.Header=Method
+RestDialog.Tab.Pagination.Title=Pagination
+RestDialog.Tab.Pagination.Info=Paginated retrieval follows the paging mode configured on your REST connection. Enable this checkbox to loop requests.\nUse the split expression to emit one Hop row per array element/node instead of full pages. Leave max pages loops blank for the Hop default safeguard (currently 256).
+RestDialog.Tab.Pagination.Enable.Label=Enable pagination loop for incoming rows (optional)
+RestDialog.Tab.Pagination.Enable.Tooltip=Runs multiple HTTP requests using the paging mode defined on REST connection metadata. When disabled, or when the connection pagination strategy is NONE, Hop performs a single request per row (legacy behaviour).
+RestDialog.Tab.Pagination.MaxPagesLoops.Label=Maximum page requests per input row (leave blank for default safeguard)
+RestDialog.Tab.Pagination.MaxPagesLoops.Tooltip=Hard iteration cap while following LINK headers, cursors, offsets, or page numbers.
+RestDialog.Tab.Pagination.SplitPath.Label=Optional result split JsonPath/XPath (XML or JSON modes)
+RestDialog.Tab.Pagination.SplitPath.Tooltip=When set with JSON or XML types, Hop emits one row per JsonPath/XPath hit. Supports '${VAR}' and '%%VAR%%'. This field skips Hop''s $[HH] hexadecimal byte decoding so bracketed JsonPath selectors (common for paging over JSON arrays) are not corrupted.
\ No newline at end of file
diff --git a/plugins/transforms/rest/src/main/samples/metadata/restconnection/github-releases.json b/plugins/transforms/rest/src/main/samples/metadata/restconnection/github-releases.json
new file mode 100644
index 0000000000..037559e1d1
--- /dev/null
+++ b/plugins/transforms/rest/src/main/samples/metadata/restconnection/github-releases.json
@@ -0,0 +1,10 @@
+{
+ "auth_type": "No Auth",
+ "base_url": "https://api.github.com",
+ "test_url": "https://api.github.com/repos/apache/hop/releases?per_page=1",
+ "ignoreSsl": false,
+ "pagination_type": "LINK_HEADER",
+ "trustStorePassword": "Encrypted ",
+ "trustStoreFile": "",
+ "name": "github-releases"
+}
diff --git a/plugins/transforms/rest/src/main/samples/metadata/restconnection/slack-conversations.json b/plugins/transforms/rest/src/main/samples/metadata/restconnection/slack-conversations.json
new file mode 100644
index 0000000000..3fb19a32d8
--- /dev/null
+++ b/plugins/transforms/rest/src/main/samples/metadata/restconnection/slack-conversations.json
@@ -0,0 +1,14 @@
+{
+ "auth_type": "No Auth",
+ "base_url": "https://slack.com",
+ "test_url": "https://slack.com/api/api.test",
+ "ignoreSsl": false,
+ "pagination_type": "BODY_CURSOR",
+ "page_param_name": "cursor",
+ "limit_param_name": "limit",
+ "default_limit": 5,
+ "cursor_json_path": "$.response_metadata.next_cursor",
+ "trustStorePassword": "Encrypted ",
+ "trustStoreFile": "",
+ "name": "slack-conversations"
+}
diff --git a/plugins/transforms/rest/src/main/samples/transforms/rest-client-github-releases-loop.hpl b/plugins/transforms/rest/src/main/samples/transforms/rest-client-github-releases-loop.hpl
new file mode 100644
index 0000000000..eeefb3deab
--- /dev/null
+++ b/plugins/transforms/rest/src/main/samples/transforms/rest-client-github-releases-loop.hpl
@@ -0,0 +1,341 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>rest-client-github-releases-loop</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+ <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+ <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2026/05/16 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/05/16 12:00:00.000</modified_date>
+ </info>
+ <notepads>
+ <notepad>
+ <backgroundcolorblue>251</backgroundcolorblue>
+ <backgroundcolorgreen>232</backgroundcolorgreen>
+ <backgroundcolorred>201</backgroundcolorred>
+ <bordercolorblue>90</bordercolorblue>
+ <bordercolorgreen>58</bordercolorgreen>
+ <bordercolorred>14</bordercolorred>
+ <fontbold>N</fontbold>
+ <fontcolorblue>90</fontcolorblue>
+ <fontcolorgreen>58</fontcolorgreen>
+ <fontcolorred>14</fontcolorred>
+ <fontitalic>N</fontitalic>
+ <fontname>SansSerif</fontname>
+ <fontsize>10</fontsize>
+ <height>168</height>
+ <width>640</width>
+ <xloc>64</xloc>
+ <yloc>32</yloc>
+ <note>Lists apache/hop GitHub releases via REST connection "github-releases" (metadata/restconnection/github-releases.json): No Auth (public API), LINK_HEADER pagination, per_page=5.
+
+About 20 releases total → ~4 HTTP pages. maxPagesLoops=8 is a safe upper bound. Requires User-Agent. Authentication is optional; a 401 usually means an invalid Bearer token was sent (e.g. unset GH_TOKEN left as ${GH_TOKEN}). For higher rate limits, edit the connection to Bearer and set a valid GitHub PAT in GH_TOKEN.
+
+Split path $[*] on the top-level JSON array returned by GET /repos/apache/hop/releases.
+</note>
+ </notepad>
+ </notepads>
+ <transform>
+ <name>generate 1 row</name>
+ <type>RowGenerator</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ </fields>
+ <interval_in_ms>5000</interval_in_ms>
+ <last_time_field>FiveSecondsAgo</last_time_field>
+ <limit>1</limit>
+ <never_ending>N</never_ending>
+ <row_time_field>now</row_time_field>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>GitHub request headers</name>
+ <type>Constant</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <field>
+ <nullif>Hop-rest-client-github-releases-loop (https://github.com/apache/hop/releases)</nullif>
+ <name>user_agent</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <nullif>application/vnd.github+json</nullif>
+ <name>accept_media</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>fetch apache-hop releases</name>
+ <type>Rest</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <applicationType>JSON</applicationType>
+ <connectionTimeout>60000</connectionTimeout>
+ <readTimeout>60000</readTimeout>
+ <connection_name>github-releases</connection_name>
+ <url>/repos/apache/hop/releases?per_page=5</url>
+ <urlInField>N</urlInField>
+ <httpPassword>Encrypted </httpPassword>
+ <preemptive>N</preemptive>
+ <method>GET</method>
+ <dynamicMethod>N</dynamicMethod>
+ <trustStorePassword>Encrypted </trustStorePassword>
+ <ignoreSsl>N</ignoreSsl>
+ <paginationEnabled>Y</paginationEnabled>
+ <maxPagesLoops>8</maxPagesLoops>
+ <resultSplitPath>$[*]</resultSplitPath>
+ <headers>
+ <header>
+ <field>user_agent</field>
+ <name>User-Agent</name>
+ </header>
+ <header>
+ <field>accept_media</field>
+ <name>Accept</name>
+ </header>
+ </headers>
+ <parameters/>
+ <matrixParameters/>
+ <result>
+ <name>release_json</name>
+ </result>
+ <retryTimes>0</retryTimes>
+ <retryDelayMs>200</retryDelayMs>
+ <retryStatusCodes/>
+ <retryMethods/>
+ <GUI>
+ <xloc>528</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>parse release JSON</name>
+ <type>JsonInput</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <include>N</include>
+ <include_field/>
+ <rownum>N</rownum>
+ <rownum_field/>
+ <limit>0</limit>
+ <valueField>release_json</valueField>
+ <IsInFields>Y</IsInFields>
+ <IsAFile>N</IsAFile>
+ <addresultfile>N</addresultfile>
+ <IsIgnoreEmptyFile>Y</IsIgnoreEmptyFile>
+ <doNotFailIfNoFile>N</doNotFailIfNoFile>
+ <ignoreMissingPath>Y</ignoreMissingPath>
+ <readurl>N</readurl>
+ <removeSourceField>Y</removeSourceField>
+ <defaultPathLeafToNull>Y</defaultPathLeafToNull>
+ <files>
+ <file>
+ <name/>
+ <filemask/>
+ <exclude_filemask/>
+ <file_required>N</file_required>
+ <include_subfolders>N</include_subfolders>
+ </file>
+ </files>
+ <fields>
+ <field>
+ <name>tag_name</name>
+ <path>$.tag_name</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>release_name</name>
+ <path>$.name</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>published_at</name>
+ <path>$.published_at</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>draft</name>
+ <path>$.draft</path>
+ <type>Boolean</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>html_url</name>
+ <path>$.html_url</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>736</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>log releases</name>
+ <type>WriteToLog</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <displayHeader>Y</displayHeader>
+ <limitRows>N</limitRows>
+ <limitRowsNumber>0</limitRowsNumber>
+ <logmessage>apache/hop GitHub release row</logmessage>
+ <loglevel>Basic</loglevel>
+ <fields>
+ <field>
+ <name>tag_name</name>
+ </field>
+ <field>
+ <name>release_name</name>
+ </field>
+ <field>
+ <name>published_at</name>
+ </field>
+ <field>
+ <name>draft</name>
+ </field>
+ <field>
+ <name>html_url</name>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>944</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <order>
+ <hop>
+ <from>generate 1 row</from>
+ <to>GitHub request headers</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>GitHub request headers</from>
+ <to>fetch apache-hop releases</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>fetch apache-hop releases</from>
+ <to>parse release JSON</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>parse release JSON</from>
+ <to>log releases</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <attributes/>
+</pipeline>
diff --git a/plugins/transforms/rest/src/main/samples/transforms/rest-client-slack-conversations-loop.hpl b/plugins/transforms/rest/src/main/samples/transforms/rest-client-slack-conversations-loop.hpl
new file mode 100644
index 0000000000..751411a334
--- /dev/null
+++ b/plugins/transforms/rest/src/main/samples/transforms/rest-client-slack-conversations-loop.hpl
@@ -0,0 +1,293 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>rest-client-slack-conversations-loop</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+ <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+ <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2026/06/06 14:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/06/06 14:00:00.000</modified_date>
+ </info>
+ <notepads>
+ <notepad>
+ <backgroundcolorblue>251</backgroundcolorblue>
+ <backgroundcolorgreen>232</backgroundcolorgreen>
+ <backgroundcolorred>201</backgroundcolorred>
+ <bordercolorblue>90</bordercolorblue>
+ <bordercolorgreen>58</bordercolorgreen>
+ <bordercolorred>14</bordercolorred>
+ <fontbold>N</fontbold>
+ <fontcolorblue>90</fontcolorblue>
+ <fontcolorgreen>58</fontcolorgreen>
+ <fontcolorred>14</fontcolorred>
+ <fontitalic>N</fontitalic>
+ <fontname>SansSerif</fontname>
+ <fontsize>10</fontsize>
+ <height>200</height>
+ <width>720</width>
+ <xloc>64</xloc>
+ <yloc>32</yloc>
+ <note>Lists Slack workspace channels via REST connection "slack-conversations" (metadata/restconnection/slack-conversations.json): BODY_CURSOR pagination on POST form body, limit=5 per page.
+
+Slack is a public API but requires a bot token. Set the SLACK_BOT_TOKEN environment/project variable (xoxb-...) with scopes channels:read or conversations:read. The static body field supplies token and types; Hop merges limit and cursor on each page.
+
+Split path $.channels[*] emits one row per channel. maxPagesLoops=10 is a safeguard.
+</note>
+ </notepad>
+ </notepads>
+ <transform>
+ <name>generate 1 row</name>
+ <type>RowGenerator</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ </fields>
+ <interval_in_ms>5000</interval_in_ms>
+ <last_time_field>FiveSecondsAgo</last_time_field>
+ <limit>1</limit>
+ <never_ending>N</never_ending>
+ <row_time_field>now</row_time_field>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>Slack POST body</name>
+ <type>Constant</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <field>
+ <nullif>token=${SLACK_BOT_TOKEN}&types=public_channel</nullif>
+ <name>request_body</name>
+ <type>String</type>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>fetch Slack channels</name>
+ <type>Rest</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <applicationType>FORM URLENCODED</applicationType>
+ <bodyField>request_body</bodyField>
+ <connectionTimeout>60000</connectionTimeout>
+ <readTimeout>60000</readTimeout>
+ <connection_name>slack-conversations</connection_name>
+ <url>/api/conversations.list</url>
+ <urlInField>N</urlInField>
+ <httpPassword>Encrypted </httpPassword>
+ <preemptive>N</preemptive>
+ <method>POST</method>
+ <dynamicMethod>N</dynamicMethod>
+ <trustStorePassword>Encrypted </trustStorePassword>
+ <ignoreSsl>N</ignoreSsl>
+ <paginationEnabled>Y</paginationEnabled>
+ <maxPagesLoops>10</maxPagesLoops>
+ <resultSplitPath>$.channels[*]</resultSplitPath>
+ <headers/>
+ <parameters/>
+ <matrixParameters/>
+ <result>
+ <name>channel_json</name>
+ </result>
+ <retryTimes>0</retryTimes>
+ <retryDelayMs>200</retryDelayMs>
+ <retryStatusCodes/>
+ <retryMethods/>
+ <GUI>
+ <xloc>528</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>parse channel JSON</name>
+ <type>JsonInput</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <include>N</include>
+ <include_field/>
+ <rownum>N</rownum>
+ <rownum_field/>
+ <limit>0</limit>
+ <valueField>channel_json</valueField>
+ <IsInFields>Y</IsInFields>
+ <IsAFile>N</IsAFile>
+ <addresultfile>N</addresultfile>
+ <IsIgnoreEmptyFile>Y</IsIgnoreEmptyFile>
+ <doNotFailIfNoFile>N</doNotFailIfNoFile>
+ <ignoreMissingPath>Y</ignoreMissingPath>
+ <readurl>N</readurl>
+ <removeSourceField>Y</removeSourceField>
+ <defaultPathLeafToNull>Y</defaultPathLeafToNull>
+ <files>
+ <file>
+ <name/>
+ <filemask/>
+ <exclude_filemask/>
+ <file_required>N</file_required>
+ <include_subfolders>N</include_subfolders>
+ </file>
+ </files>
+ <fields>
+ <field>
+ <name>channel_id</name>
+ <path>$.id</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>channel_name</name>
+ <path>$.name</path>
+ <type>String</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ <field>
+ <name>is_archived</name>
+ <path>$.is_archived</path>
+ <type>Boolean</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ <length>-1</length>
+ <repeat>N</repeat>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>736</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <transform>
+ <name>log channels</name>
+ <type>WriteToLog</type>
+ <description/>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <displayHeader>Y</displayHeader>
+ <limitRows>N</limitRows>
+ <limitRowsNumber>0</limitRowsNumber>
+ <logmessage>Slack channel row</logmessage>
+ <loglevel>Basic</loglevel>
+ <fields>
+ <field>
+ <name>channel_id</name>
+ </field>
+ <field>
+ <name>channel_name</name>
+ </field>
+ <field>
+ <name>is_archived</name>
+ </field>
+ </fields>
+ <GUI>
+ <xloc>944</xloc>
+ <yloc>240</yloc>
+ </GUI>
+ <attributes/>
+ </transform>
+ <order>
+ <hop>
+ <from>generate 1 row</from>
+ <to>Slack POST body</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Slack POST body</from>
+ <to>fetch Slack channels</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>fetch Slack channels</from>
+ <to>parse channel JSON</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>parse channel JSON</from>
+ <to>log channels</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <attributes/>
+</pipeline>
diff --git a/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPagingTest.java b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPagingTest.java
new file mode 100644
index 0000000000..42d9d83226
--- /dev/null
+++ b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/LinkHeaderPagingTest.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import jakarta.ws.rs.core.MultivaluedHashMap;
+import org.junit.jupiter.api.Test;
+
+class LinkHeaderPagingTest {
+
+ @Test
+ void prefersNextWhenPrevAndNextShareOneHeaderValue() {
+ MultivaluedHashMap<String, Object> h = new MultivaluedHashMap<>();
+ h.add(
+ "Link",
+ "<https://api.github.com/repos/apache/hop/issues?page=1>; rel=\"prev\", "
+ + "<https://api.github.com/repos/apache/hop/issues?page=3>; rel=\"next\"");
+ assertEquals(
+ "https://api.github.com/repos/apache/hop/issues?page=3",
+ LinkHeaderPaging.findFirstUriWithRelNext(h));
+ }
+
+ @Test
+ void mergesMultipleLinkHeaderFields() {
+ MultivaluedHashMap<String, Object> h = new MultivaluedHashMap<>();
+ h.add("Link", "<https://api.example.com/a>; rel=\"first\"");
+ h.add("link", "<https://api.example.com/b>; rel=\"next\"");
+ assertEquals("https://api.example.com/b", LinkHeaderPaging.findFirstUriWithRelNext(h));
+ }
+
+ @Test
+ void compoundRelTokenList() {
+ assertEquals(
+ "https://x.test/page2",
+ LinkHeaderPaging.findFirstUriWithRelNext(
+ header("<https://x.test/page2>; type=\"application/json\"; rel=\"section next\"")));
+ }
+
+ @Test
+ void unquotedRelNext() {
+ assertEquals(
+ "https://x.test/n",
+ LinkHeaderPaging.findFirstUriWithRelNext(header("<https://x.test/n>; rel=next")));
+ }
+
+ @Test
+ void commaInsideBracketsDoesNotSplitSegment() {
+ assertEquals(
+ "https://x.test/items?tags=a,b,c",
+ LinkHeaderPaging.findFirstUriWithRelNext(
+ header("<https://x.test/items?tags=a,b,c>; rel=\"next\"")));
+ }
+
+ @Test
+ void prevOnlyYieldsNull() {
+ assertNull(
+ LinkHeaderPaging.findFirstUriWithRelNext(header("<https://x.test/prev>; rel=\"prev\"")));
+ }
+
+ static MultivaluedHashMap<String, Object> header(String linkValue) {
+ MultivaluedHashMap<String, Object> h = new MultivaluedHashMap<>();
+ h.add("Link", linkValue);
+ return h;
+ }
+}
diff --git a/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMergeTest.java b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMergeTest.java
new file mode 100644
index 0000000000..15bc59856b
--- /dev/null
+++ b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingBodyMergeTest.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.LinkedHashMap;
+import org.junit.jupiter.api.Test;
+
+class PagingBodyMergeTest {
+
+ @Test
+ void mergeFormUrlEncodedAddsCursorAndPreservesExistingFields() {
+ LinkedHashMap<String, String> paging = new LinkedHashMap<>();
+ paging.put("limit", "5");
+ paging.put("cursor", "abc123");
+
+ String merged =
+ PagingBodyMerge.merge(
+ "token=xoxb-test&types=public_channel", paging, "application/x-www-form-urlencoded");
+
+ assertTrue(merged.contains("token=xoxb-test"));
+ assertTrue(merged.contains("types=public_channel"));
+ assertTrue(merged.contains("limit=5"));
+ assertTrue(merged.contains("cursor=abc123"));
+ }
+
+ @Test
+ void mergeJsonAddsNumericLimitAndCursor() {
+ LinkedHashMap<String, String> paging = new LinkedHashMap<>();
+ paging.put("MaxResults", "25");
+ paging.put("NextToken", "token-abc");
+
+ String merged = PagingBodyMerge.merge("{\"Action\":\"ListUsers\"}", paging, "application/json");
+
+ assertTrue(merged.contains("\"Action\":\"ListUsers\""));
+ assertTrue(merged.contains("\"MaxResults\":25"));
+ assertTrue(merged.contains("\"NextToken\":\"token-abc\""));
+ }
+
+ @Test
+ void parseFormUrlEncodedRoundTrip() {
+ LinkedHashMap<String, String> parsed = PagingBodyMerge.parseFormUrlEncoded("a=1&b=two%262");
+ assertEquals("1", parsed.get("a"));
+ assertEquals("two&2", parsed.get("b"));
+ }
+}
diff --git a/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingEmptySplitTest.java b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingEmptySplitTest.java
new file mode 100644
index 0000000000..2b78885c29
--- /dev/null
+++ b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingEmptySplitTest.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+import org.apache.hop.core.logging.ILoggingObject;
+import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class PagingEmptySplitTest {
+
+ private TransformMockHelper<RestMeta, RestData> mockHelper;
+ private Rest rest;
+
+ @BeforeEach
+ void setUp() {
+ mockHelper = new TransformMockHelper<>("REST empty split", RestMeta.class, RestData.class);
+ when(mockHelper.logChannelFactory.create(any(), any(ILoggingObject.class)))
+ .thenReturn(mockHelper.iLogChannel);
+
+ RestMeta meta = new RestMeta();
+ meta.setApplicationType(RestMeta.APPLICATION_TYPE_JSON);
+ RestData data = new RestData();
+ rest =
+ new Rest(
+ mockHelper.transformMeta, meta, data, 0, mockHelper.pipelineMeta, mockHelper.pipeline);
+ }
+
+ @AfterEach
+ void tearDown() {
+ mockHelper.cleanUp();
+ }
+
+ @Test
+ void shopifyStyleEmptyOrdersArrayIsLegitimate() {
+ assertTrue(
+ rest.isLegitimateEmptyPagedSplit("{\"orders\":[]}", "$.orders[*]"),
+ "Shopify 200 with no orders should not fail paging enforcement");
+ }
+
+ @Test
+ void topLevelEmptyArrayIsLegitimate() {
+ assertTrue(rest.isLegitimateEmptyPagedSplit("[]", "$[*]"));
+ }
+
+ @Test
+ void missingSplitPathIsNotLegitimate() {
+ assertFalse(rest.isLegitimateEmptyPagedSplit("{\"message\":\"Not Found\"}", "$.orders[*]"));
+ }
+}
diff --git a/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingExpressionSubstitutionTest.java b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingExpressionSubstitutionTest.java
new file mode 100644
index 0000000000..83242c65b7
--- /dev/null
+++ b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingExpressionSubstitutionTest.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hop.core.util.StringUtil;
+import org.junit.jupiter.api.Test;
+
+class PagingExpressionSubstitutionTest {
+
+ /**
+ * {@link StringUtil#environmentSubstitute} applies hex decoding and mangles JsonPath $ + bracket.
+ */
+ @Test
+ void environmentSubstituteDecodesBracketAfterDollarLikeHexNotation() {
+ String mangled =
+ StringUtil.environmentSubstitute("$[*]", Collections.singletonMap("__none__", "x"));
+ assertNotEquals("$[*]", mangled);
+ }
+
+ @Test
+ void windowsUnixOnlyPreservesJsonPathBracketStar() {
+ Map<String, String> empty = Collections.emptyMap();
+ String preserved =
+ StringUtil.substituteUnix(StringUtil.substituteWindows("$[*]", empty), empty);
+ assertEquals("$[*]", preserved);
+ }
+
+ @Test
+ void substitutesVariablesTogetherWithBracketStar() {
+ Map<String, String> m = Collections.singletonMap("SUFFIX", "&ok=1");
+ String combined =
+ StringUtil.substituteUnix(StringUtil.substituteWindows("$[*] ${SUFFIX}", m), m);
+ assertEquals("$[*] &ok=1", combined);
+ }
+}
diff --git a/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingNextUrlTest.java b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingNextUrlTest.java
new file mode 100644
index 0000000000..d618901872
--- /dev/null
+++ b/plugins/transforms/rest/src/test/java/org/apache/hop/pipeline/transforms/rest/PagingNextUrlTest.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.rest;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.jayway.jsonpath.Configuration;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.Option;
+import org.junit.jupiter.api.Test;
+
+class PagingNextUrlTest {
+
+ private static final Configuration JSON_PATH_CONFIGURATION =
+ Configuration.builder()
+ .options(Option.DEFAULT_PATH_LEAF_TO_NULL, Option.SUPPRESS_EXCEPTIONS)
+ .build();
+
+ @Test
+ void jsonPathReadsNextPageUrlFromBody() {
+ String body = "{\"data\":[{\"id\":1}],\"next\":\"https://api.example.com/items?page=2\"}";
+ Object raw = JsonPath.using(JSON_PATH_CONFIGURATION).parse(body).read("$.next");
+ assertEquals("https://api.example.com/items?page=2", raw.toString());
+ }
+
+ @Test
+ void jsonPathReturnsNullWhenNextUrlMissing() {
+ String body = "{\"data\":[]}";
+ Object raw = JsonPath.using(JSON_PATH_CONFIGURATION).parse(body).read("$.next");
+ assertTrue(raw == null);
+ }
+}
diff --git a/pom.xml b/pom.xml
index f3da4b6cd8..1bbbe2a37f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -113,6 +113,7 @@
<javaVersion>${target.jdk.version}</javaVersion>
<javadoc.failOnWarnings>false</javadoc.failOnWarnings>
<jetty.version>12.1.7</jetty.version>
+ <json-path.version>2.10.0</json-path.version>
<junit.version>6.0.3</junit.version>
<license.bundleLicenseFile>META-INF/LICENSE.txt</license.bundleLicenseFile>
<license.failOnLicenseCheck>false</license.failOnLicenseCheck>