You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Alberne (via GitHub)" <gi...@apache.org> on 2025/06/19 08:23:50 UTC
[I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Alberne opened a new issue, #9474:
URL: https://github.com/apache/seatunnel/issues/9474
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
the Http source plugin currently supports Page Number Pagination& Cursor-Based Pagination. In the following code, an NPE (NullPointerException) will occur when the pageField is null, The specific code snippet is: map.containsKey(pageField).
```
private void processPageMap(
Map<String, String> map,
String pageField,
String pageValue,
boolean usePlaceholderReplacement) {
if (usePlaceholderReplacement) {
// Placeholder replacement
Map<String, String> updatedMap = new HashMap<>();
for (Map.Entry<String, String> entry : map.entrySet()) {
String key = entry.getKey();
String value = entry.getValue();
String replacedValue = replacePlaceholder(value, pageField, pageValue);
if (replacedValue != null) {
updatedMap.put(key, replacedValue);
}
}
map.putAll(updatedMap);
// NPE (NullPointerException) will occur when the pageField is null,
} else if (map.containsKey(pageField)) {
// Key-based replacement
map.put(pageField, pageValue);
}
}
```
the processPageMap function is called in the following scenarios, only in updateRequestParam function :
```
// Process headers
if (MapUtils.isNotEmpty(this.httpParameter.getHeaders())) {
processPageMap(
this.httpParameter.getHeaders(),
pageField,
pageValue.toString(),
usePlaceholderReplacement);
processPageMap(
this.httpParameter.getHeaders(),
pageInfo.getPageCursorFieldName(),
pageInfo.getCursor(),
usePlaceholderReplacement);
}
// if not set keepPageParamAsHttpParam, but page field is in params, then set page index as
if (MapUtils.isNotEmpty(this.httpParameter.getParams())) {
processPageMap(
this.httpParameter.getParams(),
pageField,
pageValue.toString(),
usePlaceholderReplacement);
processPageMap(
this.httpParameter.getParams(),
pageInfo.getPageCursorFieldName(),
pageInfo.getCursor(),
usePlaceholderReplacement);
}
```

### SeaTunnel Version
2.3.11
### SeaTunnel Config
```conf
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Http {
plugin_output = "http"
url = "http://127.0.0.1:5000/api/cursor_data"
method = "GET"
format = "json"
content_field = "$.data.*"
keep_page_param_as_http_param = true
pageing = {
page_type="Cursor"
cursor_field ="cursor"
cursor_response_field="$.paging.cursors.next"
}
schema = {
fields {
content=string
id=int
name=string
}
}
json_field = {
content = "$.data[*].content"
id = "$.data[*].id"
name = "$.data[*].name"
}
}
}
# Console printing of the read Http data
sink {
Console {
parallelism = 1
}
}
```
### Running Command
```shell
Assuming config file is `http_paging.config`
${SEATUNNEL_HOME}/bin/seatunnel.sh --deploy-mode local -c http_paging.config
```
### Error Exception
```log
The updateRequestParam function, when called within a try...finally block’s exception handling logic, does not throw explicit exceptions.
@Override
public void internalPollNext(Collector<SeaTunnelRow> output) throws Exception {
try {
if (pageInfoOptional.isPresent()) {
noMoreElementFlag = false;
PageInfo info = pageInfoOptional.get();
// cursor pagination
if (HttpPaginationType.CURSOR.getCode().equals(info.getPageType())) {
while (!noMoreElementFlag) {
updateRequestParam(info, info.isUsePlaceholderReplacement());
pollAndCollectData(output);
Thread.sleep(10);
}
} else {
// default page number pagination
Long pageIndex = info.getPageIndex();
while (!noMoreElementFlag) {
// increment page
info.setPageIndex(pageIndex);
// set request param
updateRequestParam(info, info.isUsePlaceholderReplacement());
pollAndCollectData(output);
pageIndex += 1;
Thread.sleep(10);
}
}
} else {
pollAndCollectData(output);
}
} finally {
if (Boundedness.BOUNDED.equals(context.getBoundedness()) && noMoreElementFlag) {
// signal to the source that we have reached the end of the data.
log.info("Closed the bounded http source");
context.signalNoMoreElement();
} else {
if (httpParameter.getPollIntervalMillis() > 0) {
Thread.sleep(httpParameter.getPollIntervalMillis());
}
}
}
}
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-3141758609
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9474: [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null
URL: https://github.com/apache/seatunnel/issues/9474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-3115418808
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "Alberne (via GitHub)" <gi...@apache.org>.
Alberne commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-2998882967
@Hisoka-X please assign to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "Alberne (via GitHub)" <gi...@apache.org>.
Alberne commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-2995024974
@hailin0 please assign to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "Alberne (via GitHub)" <gi...@apache.org>.
Alberne commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-2987208430
1. Page Number Pagination
```
processPageMap(
this.httpParameter.getParams(),
pageField,
pageValue.toString(),
usePlaceholderReplacement);
```
2. Cursor-Based Pagination
```
processPageMap(
this.httpParameter.getParams(),
pageInfo.getPageCursorFieldName(),
pageInfo.getCursor(),
usePlaceholderReplacement);
```
The above code logic exists simultaneously within the updateRequestParam function, but in practice only one pagination method can be selected, Page Number Pagination or Cursor-Based Pagination .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [seatunnel-connector-http-base] An NPE (NullPointerException) will occur when the pageField is null [seatunnel]
Posted by "Alberne (via GitHub)" <gi...@apache.org>.
Alberne commented on issue #9474:
URL: https://github.com/apache/seatunnel/issues/9474#issuecomment-2987187610
please assign to me~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org