You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "hefeng199010 (via GitHub)" <gi...@apache.org> on 2025/07/17 09:16:55 UTC
[I] [Bug] [transform-v2] 加上transform table_filter后就报错 [seatunnel]
hefeng199010 opened a new issue, #9583:
URL: https://github.com/apache/seatunnel/issues/9583
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
<img width="1397" height="868" alt="Image" src="https://github.com/user-attachments/assets/4fbfb4b2-8e70-4107-b271-b13f13fe61f7" />
加上transform table_filter后就报错,
<img width="2403" height="1148" alt="Image" src="https://github.com/user-attachments/assets/07c3343b-c3a2-45c0-871b-0922245b2658" />
经过调试源码,发现加了transform后,sink就不在map里面
<img width="1560" height="753" alt="Image" src="https://github.com/user-attachments/assets/cef8797f-5599-4b10-afd9-a11ab27f81ab" />
### SeaTunnel Version
目前数影seatunnel2.3.11
### SeaTunnel Config
```conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of streaming processing in SeaTunnel config
######
env {
# You can set SeaTunnel environment configuration here
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
read_limit.bytes_per_second=70000000
read_limit.rows_per_second=100000
}
source {
MySQL-CDC {
server-id = 11
username = "root"
password = "root%123"
database-names = [""]
table-pattern = "test2.*\\.*.*"
base-url = "jdbc:mysql://hadoop001:3306/test2"
startup.mode="INITIAL"
snapshot.split.size = 8096
snapshot.fetch.size=4096
server-time-zone="Asia/Shanghai"
exactly_once=true
schema-changes.enabled = true
plugin_output = "source1"
}
}
transform {
TableFilter {
plugin_input = "source1"
plugin_output = "transform_1"
database_pattern = "test2"
table_pattern = "test2.student"
pattern_mode = "EXCLUDE"
}
}
sink {
StarRocks {
plugin_input = "transform_1"
nodeUrls = []
base-url = "jdbc:mysql://xx:xxx"
username = xxx
password = "xxx"
database = "xxx"
max_retries=10
batch_max_rows = 100000
schema_save_mode="CREATE_SCHEMA_WHEN_NOT_EXIST"
starrocks.config = {
format = "JSON"
strip_outer_array = true
}
enable_upsert_delete = true
save_mode_create_template = """
CREATE TABLE IF NOT EXISTS xxx.`${table_name}` (
${rowtype_primary_key},
${rowtype_fields}
) ENGINE=OLAP
PRIMARY KEY (${rowtype_primary_key})
DISTRIBUTED BY HASH (${rowtype_primary_key})
PROPERTIES (
"replication_num" = "2",
"in_memory" = "false",
"enable_persistent_index" = "true",
"replicated_storage" = "true",
"compression" = "LZ4"
)
"""
}
}
```
### Running Command
```shell
idea本地运行
```
### Error Exception
```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.example.engine.SeaTunnelEngineLocalExample.main(SeaTunnelEngineLocalExample.java:49)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.tryGenerateMultiTableSink(MultipleTableJobConfigParser.java:642)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:605)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:240)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "nankeChen75 (via GitHub)" <gi...@apache.org>.
nankeChen75 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-5971922995
I investigated and reproduced this issue on SeaTunnel 2.3.11.
The failure occurs when `TableFilter` filters out all input tables. In that case, `sinkActions` becomes empty, but `tryGenerateMultiTableSink()` still accesses `sinkActions.get(0)`, which causes the `IndexOutOfBoundsException`.
This specific exception appears to have been addressed by #10600:
https://github.com/apache/seatunnel/commit/2739847410
The following guard is present in current `dev` and v3.0.0:
```java
if (sinkActions.isEmpty()) {
return Optional.empty();
}
```
I also checked v2.3.13, where this guard is not present, so the 2.3.x release line still appears to be affected by this specific failure.
Could the maintainers confirm whether this issue can be considered fixed in v3.0.0?
Separately, should this guard be backported to the 2.3.x branch, or should the issue remain open for that release line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] 加上transform table_filter后就报错 [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-3194005178
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-5975911138
Thanks for the report, and thanks @nankeChen75 for digging into it.
From the stack trace, the `IndexOutOfBoundsException: Index: 0, Size: 0` is thrown in `MultipleTableJobConfigParser.tryGenerateMultiTableSink` on 2.3.11, and the config has a `TableFilter` transform in `EXCLUDE` mode between the MySQL-CDC source and the StarRocks sink. @nankeChen75's analysis is that the filter leaves the sink with no upstream tables, so `sinkActions` is empty when `get(0)` is called, which matches the observation that the sink is not in the map.
@hefeng199010, to confirm that reading, could you share:
- which tables the source `table-pattern` actually matched in your run (how many, and their names)?
- whether the job runs if you remove the `TableFilter` transform, or change the filter so that at least one table is kept?
Regarding the fix status and whether a backport to 2.3.x is needed, I'd like to verify against the code before answering; I'll follow up once I've checked the change referenced in #10600.
<!-- streview-comment:1502 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "nankeChen75 (via GitHub)" <gi...@apache.org>.
nankeChen75 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-6009476423
The minimal backport based on `2.3.13-release` has already been implemented in #12638.
The PR also already includes regression coverage for the all-tables-filtered case, including both a parser unit test and a Zeta integration test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "nankeChen75 (via GitHub)" <gi...@apache.org>.
nankeChen75 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-5994115155
Thanks for the guidance. I have completed the reproduction checks and prepared a minimal backport based on `2.3.13-release`.
**Reproduction on 2.3.11**
Using a minimal FakeSource → TableFilter (EXCLUDE) → Console job, filtering out every table causes the job to exit with code 1:
```
org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40)
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
at java.base/java.util.Objects.checkIndex(Objects.java:361)
at java.base/java.util.ArrayList.get(ArrayList.java:427)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.tryGenerateMultiTableSink(MultipleTableJobConfigParser.java:642)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:605)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:240)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
... 2 more
```
The complete credential-free reproduction configuration is:
```
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
plugin_output = "source1"
tables_configs = [
{
row.num = 1
schema = {
table = "test.student"
columns = [
{ name = "id", type = "int" }
]
}
}
]
}
}
transform {
TableFilter {
plugin_input = "source1"
plugin_output = "filtered"
database_pattern = "test"
table_pattern = "student"
pattern_mode = "EXCLUDE"
}
}
sink {
console {
plugin_input = "filtered"
}
}
```
**Partial filtering**
I also tested three source tables: student (1 row), teacher (3 rows), and course (2 rows). Using only `table_pattern = "student"` with
`pattern_mode = "EXCLUDE"` leaves teacher and course.
The logs confirm that the execution plan contains `console-MultiTableSink`, that `MultiTableSinkWriter` is initialized, and that Console writes exactly 3 teacher rows and 2 course rows. The job finishes with `FINISHED`.
**Backport and regression coverage**
The original fix is part of a much larger table-level fault-isolation change, so I did not cherry-pick the whole commit. Instead, I backported only the minimal `sinkActions.isEmpty()` guard relevant to this issue.
I added:
- A parser unit test asserting that filtering all tables produces no sink actions.
- A Zeta integration test with an all-filtered branch alongside a retained multi-table branch. The assertions verify the exact row counts for the retained tables.
Both new tests fail with `IndexOutOfBoundsException` when the guard is removed and pass when it is restored. For the integration test, I rebuilt the starter JAR before each run to verify the actual container-executed code.
Formatting and build verification have also completed successfully.
These results cover the empty-sinkActions failure and partial filtering in the minimal reproduction; they do not establish that every symptom
in the original MySQL-CDC → StarRocks report has the same cause.
I will open the backport PR and link it here. Please let me know if `2.3.13-release` is not the intended target branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-6008428831
Thanks @nankeChen75 for the investigation and the reproduction. The analysis matches: when `TableFilter` filters out every table, `sinkActions` is empty and `tryGenerateMultiTableSink()` still calls `sinkActions.get(0)`, which triggers the `IndexOutOfBoundsException`.
On your two questions:
1. **v3.0.0**: based on your check that the empty-`sinkActions` guard from 2739847410 is present in current `dev` and v3.0.0, this specific failure can be considered fixed for that release line.
2. **2.3.x backport**: since you confirmed the guard is absent in v2.3.13, the 2.3.x line is still affected. A minimal backport based on `2.3.13-release` sounds like the right approach, and a PR for it is welcome. Please keep the change scoped to the guard itself, add a test covering the "all tables filtered out" case (your reproduction config is a good basis), and reference this issue and 2739847410 in the PR description.
Let's keep this issue open to track the 2.3.x fix until the backport lands.
<!-- streview-comment:1546 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-6030199671
Thanks for the thorough follow-up, @nankeChen75. The reproduction on 2.3.11 and the stack trace line up with your analysis: `TableFilter` drops every table, `sinkActions` ends up empty, and `tryGenerateMultiTableSink()` still calls `sinkActions.get(0)`.
Good to hear the minimal backport based on `2.3.13-release` is already up and includes both a parser unit test and a Zeta integration test for the all-tables-filtered case.
Two asks to help move the backport along:
1. Please keep the backport scoped to the `sinkActions.isEmpty()` guard from 2739847410 plus the tests. If you had to deviate from that change for the 2.3.x code base, please call it out in the PR description.
2. Could you confirm that both new tests fail on `2.3.13-release` without the guard and pass with it, so they actually pin the regression?
Once those are in place I'll take a look at the PR. Since you noted the guard is already present in `dev` and v3.0.0, this issue can stay open to track the 2.3.x backport.
<!-- streview-comment:1574 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-5988336955
Thanks @nankeChen75 for digging into this and reproducing it on 2.3.11.
Your analysis matches what the original report hinted at: when `TableFilter` drops every input table, `sinkActions` ends up empty and `tryGenerateMultiTableSink()` still calls `sinkActions.get(0)`, hence the `IndexOutOfBoundsException`. The `sinkActions.isEmpty()` guard you quoted from 2739847410 is the right fix for that path.
On your two questions:
1. **Fixed in v3.0.0?** For the "all tables filtered out → IndexOutOfBounds" failure, yes, the guard you found covers it. What I can't confirm yet is whether that is the *only* failure behind this issue: the original report only had the error text and full config in screenshots / truncated form, and it described the sink being missing from an internal table map, which may or may not be the same code path. If you still have your repro, could you paste the full stack trace as text (and the complete config, with credentials redacted) so we can verify it is the same exception and close this with confidence?
2. **Backport to 2.3.x?** Since you've checked that the guard is absent in v2.3.13, a backport makes sense. If you're willing, please open a PR against the 2.3.x branch cherry-picking the guard from 2739847410 (plus a small unit test for the empty-`sinkActions` case if feasible) and link it here. We'll keep this issue open until that lands or until we decide the 2.3.x line won't receive it.
One more small ask: in your repro, does the job still behave correctly when `TableFilter` keeps *some* but not all tables (i.e. the multi-table sink is generated with a reduced table set)? That would help us rule out the "sink missing from table map" variant being a separate bug.
<!-- streview-comment:1534 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "goutamadwant (via GitHub) via commits" <co...@seatunnel.apache.org>.
goutamadwant closed issue #9583: [Bug] [transform-v2] An error occurs after adding transform table_filter
URL: https://github.com/apache/seatunnel/issues/9583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [transform-v2] An error occurs after adding transform table_filter [seatunnel]
Posted by "goutamadwant (via GitHub) via commits" <co...@seatunnel.apache.org>.
goutamadwant commented on issue #9583:
URL: https://github.com/apache/seatunnel/issues/9583#issuecomment-6651172350
The guard landed in #10600; closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org