You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kwontaeheon (via GitHub)" <gi...@apache.org> on 2024/09/05 12:39:27 UTC

[I] [Feature] Kafka CDC type conversion [paimon]

kwontaeheon opened a new issue, #4134:
URL: https://github.com/apache/paimon/issues/4134

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   Using Maxwell CDC action, complex types can not changed to string.
   Can I define the type of kafka source from commandline or, not to cast to string?
   the samples are below.
   
   * paimon table (created from spark)
   ```sql
   create table if not exists `local`.`default`.`id_mapping`
   ( id STRING, val_list ARRAY<STRING>, primary key (id) NOT ENFORCED));
   ```
   
   
   * kafka message json
   ```json
   {"type": "insert", "primary_key": ["1111"], "primary_key_columns": ["id"], "data": {"id": "1111", "val_list": ["2", "3"]}}
   ```
   
   * error messages
   ```
   java.lang.UnsupportedOperationException: Cannot convert field val_list from type ARRAY<STRING> to STRING of Paimon table default.id_mapping.
   	at org.apache.paimon.flink.sink.cdc.UpdatedDataFieldsProcessFunctionBase.applySchemaChange(UpdatedDataFieldsProcessFunctionBase.java:124)
   	at org.apache.paimon.flink.sink.cdc.UpdatedDataFieldsProcessFunction.processElement(UpdatedDataFieldsProcessFunction.java:58)
   	at org.apache.paimon.flink.sink.cdc.UpdatedDataFieldsProcessFunction.processElement(UpdatedDataFieldsProcessFunction.java:39)
   	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
   	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:238)
   	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:157)
   	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:114)
   	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:579)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
   	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
   	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   ```
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Kafka CDC type conversion [paimon]

Posted by "kwontaeheon (via GitHub)" <gi...@apache.org>.
kwontaeheon commented on issue #4134:
URL: https://github.com/apache/paimon/issues/4134#issuecomment-2333380247

   Without paimon flink action, I found I can easily ingest cdc by flink table api.
   Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Kafka CDC type conversion [paimon]

Posted by "kwontaeheon (via GitHub)" <gi...@apache.org>.
kwontaeheon closed issue #4134: [Feature] Kafka CDC type conversion
URL: https://github.com/apache/paimon/issues/4134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org