You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "krutoileshii (via GitHub)" <gi...@apache.org> on 2026/01/10 03:37:39 UTC
[I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
krutoileshii opened a new issue, #10317:
URL: https://github.com/apache/seatunnel/issues/10317
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When adding the FieldRename transform to the job config, I start getting errors for my Iceberg Sink. This is a CDC Source.
If i remove the FieldRename, the sink works perfectly with no issues.
The source tables has the following columns:
- InvoiceNum
- InvoiceLine
- VendorNum
- VendorID
- Amount
```
############################################################ # vendorroyalties_cdc_to_iceberg_polaris_upsert.conf ############################################################
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 10000
checkpoint.timeout = 600000
}
source {
SqlServer-CDC {
plugin_output = "mssql_cdc"
username = "user_name"
password = "password"
url = "jdbc:sqlserver://IP:PORT;databaseName=db_name;encrypt=true;trustServerCertificate=true"
database-names = ["AlloCustom"]
table-names = ["AlloCustom.Finance.VendorRoyalties"]
startup.mode = "initial"
debezium {
database.encrypt = "true"
database.trustServerCertificate = "true"
}
incremental.parallelism = 1
snapshot.split.size = 8096
}
}
transform {
Metadata {
metadata_fields {
Database = database
Table = table
RowKind = rowKind
EventTime = ts_ms
Delay = delay
}
plugin_input = "mssql_cdc"
plugin_output = "add_metadata"
}
TableRename {
plugin_input = "add_metadata"
plugin_output = "table_names_to_snake_case"
replacements_with_regex = [
# Normalize separators (kebab, spaces, dots, $, etc.) -> underscore
{ replace_from = "[^A-Za-z0-9_]+", replace_to = "_" }, # Insert underscores at boundaries (no capture groups needed)
{ replace_from = "(?<=[A-Z])(?=[A-Z][a-z])", replace_to = "_" }, # HTTPResponse -> HTTP_Response
{ replace_from = "(?<=[a-z0-9])(?=[A-Z])", replace_to = "_" }, # camelCase / 500Error
{ replace_from = "(?<=[A-Za-z])(?=[0-9])", replace_to = "_" }, # version2 -> version_2
{ replace_from = "(?<=[0-9])(?=[A-Za-z])", replace_to = "_" } # user123Name -> user_123_Name
]
convert_case = "LOWER"
}
FieldRename {
plugin_input = "table_names_to_snake_case"
plugin_output = "column_names_to_snake_case"
replacements_with_regex = [
{ replace_from = "[^A-Za-z0-9_]+", replace_to = "_" },
{ replace_from = "(?<=[A-Z])(?=[A-Z][a-z])", replace_to = "_" },
{ replace_from = "(?<=[a-z0-9])(?=[A-Z])", replace_to = "_" },
{ replace_from = "(?<=[A-Za-z])(?=[0-9])", replace_to = "_" },
{ replace_from = "(?<=[0-9])(?=[A-Za-z])", replace_to = "_" }
]
convert_case = "LOWER"
}
}
sink {
Iceberg {
plugin_input = "column_names_to_snake_case"
catalog_name = "polaris"
namespace = "allocustom"
table = "${schema_name}__${table_name}"
# Upsert mode for CDC correctness
iceberg.table.upsert-mode-enabled = true
# Best-effort composite key (edit if you know the real unique key)
iceberg.table.primary-keys = "invoice_num,invoice_line,vendor_num,vendor_id"
iceberg.table.schema-evolution-enabled = true
iceberg.table.write-props = {
write.format.default = "parquet"
}
iceberg.catalog.config = {
"type" = "rest"
"uri" = "http://ip:8181/api/catalog"
"warehouse" = "data-warehouse-test"
"credential" = "admin:password"
"scope" = "PRINCIPAL_ROLE:ALL"
"token-refresh-enabled" = "true"
"io-impl" = "org.apache.iceberg.aws.s3.S3FileIO"
"warehouse.location" = "s3://testdec16/raw"
"client.region" = "us-east-1"
"s3.endpoint" = "https://endpoint"
"s3.path-style-access" = "true"
"s3.access-key-id" = "access_key"
"s3.secret-access-key" = "secret_key"
}
}
}
```
### SeaTunnel Version
v2.3.12
### SeaTunnel Config
```conf
Default out of the box.
```
### Running Command
```shell
./bin/seatunnel.sh --config test.conf
```
### Error Exception
```log
2026-01-09 15:53:20,084 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Iceberg' with upstream input catalog-table[database: allocustom, schema: finance, table: vendor_royalties] 2026-01-09 15:53:20,137 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Start submit job, job id: 1062142054877888513, with plugin jar [file:/opt/seatunnel/apache-seatunnel-2.3.12/connectors/connector-iceberg-2.3.12.jar, file:/opt/seatunnel/apache-seatunnel-2.3.12/connectors/connector-cdc-sqlserver-2.3.12.jar, file:/opt/seatunnel/apache-seatunnel-2.3.12/connectors/connector-cdc-base-2.3.12.jar] 2026-01-09 15:53:20,209 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2026-01-09 15:53:20,211 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [127.0.0.1]:5801:6e348755-fc5a-4dbd-842b-2ea14c5fe57b, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel
{/127.0.0.1:49585->/127.0.0.1:5801}, remoteAddress=[127.0.0.1]:5801, lastReadTime=2026-01-09 15:53:20.204, lastWriteTime=2026-01-09 15:53:20.139, closedTime=2026-01-09 15:53:20.210, connected server version=5.1} 2026-01-09 15:53:20,212 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [127.0.0.1]:5802:db82fd2f-d216-4ed3-bbc8-bcfad09b9089, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/127.0.0.1:39665->/127.0.0.1:5802}, remoteAddress=[127.0.0.1]:5802, lastReadTime=2026-01-09 15:53:19.528, lastWriteTime=2026-01-09 15:53:19.527, closedTime=2026-01-09 15:53:20.211, connected server version=5.1} 2026-01-09 15:53:20,212 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2026-01-09 15:53:20,214 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN 2026
-01-09 15:53:20,215 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client...... 2026-01-09 15:53:20,215 ERROR [o.a.s.c.s.SeaTunnel ] [main] - =============================================================================== 2026-01-09 15:53:20,215 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error, 2026-01-09 15:53:20,215 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues 2026-01-09 15:53:20,215 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed 2026-01-09 15:53:20,216 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.
java:40) Caused by: java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed] at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:569) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:575) at org.apache.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:267) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto
(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:261) at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:663) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.
util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.lambda$toIcebergSchema$6(SchemaUtils.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.toIcebergSchema(SchemaUtils.java:
178) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.autoCreateTable(SchemaUtils.java:106) at org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.createTable(IcebergCatalog.java:180) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createTable(DefaultSaveModeHandler.java:213) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:139) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandler.java:88) at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:42) at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:564) ... 21 more at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347) at com.hazelcast.spi.impl.AbstractInvocationFu
ture.cascadeException(AbstractInvocationFuture.java:1340) at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65) at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759) at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1235) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:
321) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:251) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135) at com.hazelcast.client.impl.protocol.
util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89) at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed] at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:569) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:575) at org.apach
e.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:267) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forE
ach(ReferencePipeline.java:497) at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:261) at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:663) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.lambda$toIcebergSchema$6(SchemaUtils.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaini
ng(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.toIcebergSchema(SchemaUtils.java:178) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.autoCreateTable(SchemaUtils.java:106) at org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.createTable(IcebergCatalog.java:180) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createTable(DefaultSaveModeHandler.java:213) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:139)
at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandler.java:88) at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:42) at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:564) ... 21 more at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:670) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) 2026-01-09 15:53:20,217 ERROR [o.a.s.c.s.SeaTunnel ] [main] - ==================
============================================================= Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:228) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:40) Caused by: java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed] at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:569) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:575) at org.apache.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:267) at java.base/java.util
.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMast
er.java:261) at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:663) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.lambda$toIcebergSchema$6(SchemaUtils.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.ja
va:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.toIcebergSchema(SchemaUtils.java:178) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.autoCreateTable(SchemaUtils.java:106) at org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.createTable(IcebergCatalog.java:180) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createTable(DefaultSaveModeHandler.java:213) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:139) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandle
r.java:88) at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:42) at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:564) ... 21 more at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347) at com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340) at com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65) at com.hazelcast.spi.impl.AbstractInvocationFuture$ApplyNode.execute(AbstractInvocationFuture.java:1478) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockOtherNode(AbstractInvocationFuture.java:797) at com.hazelcast.spi.impl.AbstractInvocationFuture.unblockAll(AbstractInvocationFuture.java:759) at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFu
ture.java:1235) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223) at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionally(AbstractInvocationFuture.java:709) at com.hazelcast.client.impl.spi.impl.ClientInvocation.completeExceptionally(ClientInvocation.java:294) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:321) at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyException(ClientInvocation.java:304) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.handleResponse(ClientResponseHandlerSupplier.java:164) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.process(ClientResponseHandlerSupplier.java:141) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier.access$300(ClientResponseHandlerSupplier.java:60) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicRespo
nseHandler.accept(ClientResponseHandlerSupplier.java:251) at com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$DynamicResponseHandler.accept(ClientResponseHandlerSupplier.java:243) at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.handleClientMessage(TcpClientConnection.java:245) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.handleMessage(ClientMessageDecoder.java:135) at com.hazelcast.client.impl.protocol.util.ClientMessageDecoder.onRead(ClientMessageDecoder.java:89) at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:136) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) at com.hazelcast.internal.util.ex
ecutor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: org.apache.seatunnel.engine.common.exception.JobException: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed] at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:569) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:575) at org.apache.seatunnel.engine.server.master.JobMaster.lambda$init$1(JobMaster.java:267) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractP
ipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:261) at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:663) at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.lambda$toIcebergSchema$6(SchemaUtils.java:177) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.toIcebergSchema(S
chemaUtils.java:178) at org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils.autoCreateTable(SchemaUtils.java:106) at org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.createTable(IcebergCatalog.java:180) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createTable(DefaultSaveModeHandler.java:213) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:139) at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandler.java:88) at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:42) at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36) at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:564) ... 21 more at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$6(CoordinatorService.java:670) at org.apache.seatunnel.api.tracing.MD
CRunnable.run(MDCRunnable.java:43) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834)
```
### Zeta or Flink or Spark Version
Zeta Engine
### Java or Scala Version
Java 11
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10317:
URL: https://github.com/apache/seatunnel/issues/10317#issuecomment-3739890549
@yzeng1618 Thank you adding is_regex got it working.
Do have a question, if i have multiple tables coming from the source, is there a way to specify multiple primary key combinations for each one as they could be different depending on the table? or do i have to define multiple syncs for this?
```
############################################################ # vendorroyalties_cdc_to_iceberg_polaris_upsert.conf ############################################################
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 10000
checkpoint.timeout = 600000
}
source {
SqlServer-CDC {
plugin_output = "mssql_cdc"
username = "user_name"
password = "password"
url = "jdbc:sqlserver://IP:PORT;databaseName=db_name;encrypt=true;trustServerCertificate=true"
database-names = ["AlloCustom"]
table-names = ["AlloCustom.Finance.VendorRoyalties","AlloCustom.Finance.Parts"]
startup.mode = "initial"
debezium {
database.encrypt = "true"
database.trustServerCertificate = "true"
}
incremental.parallelism = 1
snapshot.split.size = 8096
}
}
transform {
Metadata {
metadata_fields {
Database = database
Table = table
RowKind = rowKind
EventTime = ts_ms
Delay = delay
}
plugin_input = "mssql_cdc"
plugin_output = "add_metadata"
}
TableRename {
plugin_input = "add_metadata"
plugin_output = "table_names_to_snake_case"
replacements_with_regex = [
# Normalize separators (kebab, spaces, dots, $, etc.) -> underscore
{ replace_from = "[^A-Za-z0-9_]+", replace_to = "_" }, # Insert underscores at boundaries (no capture groups needed)
{ replace_from = "(?<=[A-Z])(?=[A-Z][a-z])", replace_to = "_" }, # HTTPResponse -> HTTP_Response
{ replace_from = "(?<=[a-z0-9])(?=[A-Z])", replace_to = "_" }, # camelCase / 500Error
{ replace_from = "(?<=[A-Za-z])(?=[0-9])", replace_to = "_" }, # version2 -> version_2
{ replace_from = "(?<=[0-9])(?=[A-Za-z])", replace_to = "_" } # user123Name -> user_123_Name
]
convert_case = "LOWER"
}
FieldRename {
plugin_input = "table_names_to_snake_case"
plugin_output = "column_names_to_snake_case"
replacements_with_regex = [
{ replace_from = "[^A-Za-z0-9_]+", replace_to = "_" },
{ replace_from = "(?<=[A-Z])(?=[A-Z][a-z])", replace_to = "_" },
{ replace_from = "(?<=[a-z0-9])(?=[A-Z])", replace_to = "_" },
{ replace_from = "(?<=[A-Za-z])(?=[0-9])", replace_to = "_" },
{ replace_from = "(?<=[0-9])(?=[A-Za-z])", replace_to = "_" }
]
convert_case = "LOWER"
}
}
sink {
Iceberg {
plugin_input = "column_names_to_snake_case"
catalog_name = "polaris"
namespace = "allocustom"
table = "${schema_name}__${table_name}"
# Upsert mode for CDC correctness
iceberg.table.upsert-mode-enabled = true
# Best-effort composite key (edit if you know the real unique key)
iceberg.table.primary-keys = [
{ "vendor_royalties" = "invoice_num,invoice_line,vendor_num,vendor_id"},
{"parts" = "part_num,company,vendor_num"}
]
iceberg.table.schema-evolution-enabled = true
iceberg.table.write-props = {
write.format.default = "parquet"
}
iceberg.catalog.config = {
"type" = "rest"
"uri" = "http://ip:8181/api/catalog"
"warehouse" = "data-warehouse-test"
"credential" = "admin:password"
"scope" = "PRINCIPAL_ROLE:ALL"
"token-refresh-enabled" = "true"
"io-impl" = "org.apache.iceberg.aws.s3.S3FileIO"
"warehouse.location" = "s3://testdec16/raw"
"client.region" = "us-east-1"
"s3.endpoint" = "https://endpoint"
"s3.path-style-access" = "true"
"s3.access-key-id" = "access_key"
"s3.secret-access-key" = "secret_key"
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10317:
URL: https://github.com/apache/seatunnel/issues/10317#issuecomment-3736674399
<img width="973" height="356" alt="Image" src="https://github.com/user-attachments/assets/428a6d5f-cedc-4385-af6c-b90620667130" />
1. Why it fails only when FieldRename is enabled
- The Iceberg sink resolves identifier fields from iceberg.table.primary-keys during table creation (SchemaUtils.toIcebergSchema in connector-iceberg).
- If any configured PK field name does not exist in the incoming schema, structType.field(pk) becomes null and the current code throws an NPE.
- In FieldRename, replacements_with_regex is only treated as regex when is_regex = true is set for each rule. If is_regex is omitted (null), it’s treated as a literal full-match, so patterns like (?<=[a-z0-9])(?=[A-Z]) never match.
- Result: columns may become invoicenum/vendorid (lowercase only) instead of invoice_num/vendor_id, so the PK list can’t be resolved and Iceberg NPEs.
2. Workaround
- Add is_regex = true to each FieldRename.replacements_with_regex item, e.g. { replace_from = "(?<=[a-z0-9])(?=[A-Z])", replace_to = "_", is_regex = true }
- As a quick validation, you can also temporarily change iceberg.table.primary-keys to match the actual output column names after FieldRename.
3. Follow-up / fix plan
- Make FieldRename default is_regex to true (align behavior with docs and TableRename) and update the docs accordingly.
- Improve the Iceberg sink to fail fast with a clear “missing PK field(s)” error instead of throwing an NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10317:
URL: https://github.com/apache/seatunnel/issues/10317#issuecomment-3741373146
Your current configuration is incorrect. The `iceberg.table.primary-keys` of Iceberg sink is a **single string** option that applies to all tables, and it does not support the per-table configuration format you are attempting to use.
The correct approach is to use the `table-names-config` option in the **SqlServer-CDC source** to specify different primary keys for each table:
Modify your configuration as follows:
```hocon
source {
SqlServer-CDC {
plugin_output = "mssql_cdc"
username = "user_name"
password = "password"
url = "jdbc:sqlserver://IP:PORT;databaseName=db_name;encrypt=true;trustServerCertificate=true"
database-names = ["AlloCustom"]
table-names = ["AlloCustom.Finance.VendorRoyalties","AlloCustom.Finance.Parts"]
# Specify different primary keys for each table
table-names-config = [
{
table = "AlloCustom.Finance.VendorRoyalties"
primaryKeys = ["invoice_num", "invoice_line", "vendor_num", "vendor_id"]
},
{
table = "AlloCustom.Finance.Parts"
primaryKeys = ["part_num", "company", "vendor_num"]
}
]
startup.mode = "initial"
debezium {
database.encrypt = "true"
database.trustServerCertificate = "true"
}
incremental.parallelism = 1
snapshot.split.size = 8096
}
}
``` [3](#0-2)
Then **remove** the `iceberg.table.primary-keys` configuration from the Iceberg sink:
```hocon
sink {
Iceberg {
plugin_input = "column_names_to_snake_case"
catalog_name = "polaris"
namespace = "allocustom"
table = "${schema_name}__${table_name}"
iceberg.table.upsert-mode-enabled = true
# Remove the iceberg.table.primary-keys configuration
# Iceberg will automatically inherit primary keys from the source table schema
iceberg.table.schema-evolution-enabled = true
# ... Other configurations
}
}
```
You can try the modified content above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10317:
URL: https://github.com/apache/seatunnel/issues/10317#issuecomment-3741386009
@yzeng1618
Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink. [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 closed issue #10317: [Bug] [FieldRename+Iceberg+Zeta] Adding FieldRename transform breaks Iceberg Sink.
URL: https://github.com/apache/seatunnel/issues/10317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org