You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "klyashko (via GitHub)" <gi...@apache.org> on 2025/01/30 15:11:05 UTC

[I] [Bug] Apache Paimon schema evolution [paimon]

klyashko opened a new issue, #5004:
URL: https://github.com/apache/paimon/issues/5004

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   Not sure that it's a bag, but wanted to clarify the question of schema evolution.
   
   Currently we are onboarding Paimon in way:
   1. There is a Flink application which injects data to Paimon tables
   2. There are readers which consumes the data later
   
   During experimenting with schema evolution noticed that when schema change was done in external way Flink application didn't pick up changed properties even after it was restarted. For application to pick up the change I had to restart job without its state. What riese a question how to properly manage the schema evolution, should that be imbedded to the application as pre job execution steps or should that be managed somehow else?
   
   ### Compute Engine
   
   Flink
   
   ### Minimal reproduce step
   
   1. Write to Paimon table from stream application in way:
   
   `  
       val table = env.sqlQuery(query)
       val data = env.toChangelogStream(table)
   
       val catalog = FlinkCatalogFactory.createPaimonCatalog(...)
       val pTable = catalog.getTable(Identifier.create(..., ...))
   
       new FlinkSinkBuilder(pTable)
         .forRow(data, table.getSchema.toRowDataType)
         .parallelism(1)
         .build()
   `
   Usage of `HiveCatalog` didn't change the behaviour.
   
   2. Change Paimon table property with external Java application based on Catalog API documentation section
   
   ### What doesn't meet your expectations?
   
   Absence of understanding after consulting with documentation how safely evolve table schema. In our case it was attempt to change `snapshot.time-retained`. And preferably do it without losing job state
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Apache Paimon schema evolution [paimon]

Posted by "dahai1996 (via GitHub)" <gi...@apache.org>.
dahai1996 commented on issue #5004:
URL: https://github.com/apache/paimon/issues/5004#issuecomment-2696151315

   `table.getSchema.toRowDataType` run on jobmanager,so you need restart flink,try this : 
   ```
   RichCdcSinkBuilder
   ```
   see : https://paimon.apache.org/docs/master/program-api/flink-api/#cdc-ingestion-table


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Apache Paimon schema evolution [paimon]

Posted by "liyubin117 (via GitHub)" <gi...@apache.org>.
liyubin117 commented on issue #5004:
URL: https://github.com/apache/paimon/issues/5004#issuecomment-2671305616

   In our scenarios, we use `alter table set` sql to change `consumer.expiration-time` without restarting jobs and make it take effect. 
   Notablely, paimon version: 0.9.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Apache Paimon schema evolution [paimon]

Posted by "klyashko (via GitHub)" <gi...@apache.org>.
klyashko commented on issue #5004:
URL: https://github.com/apache/paimon/issues/5004#issuecomment-2673822727

   @liyubin117 thanks for your reply. Do you do anything extra for change to take an effect? Or how much time need to pass for job to notice the change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Apache Paimon schema evolution [paimon]

Posted by "liyubin117 (via GitHub)" <gi...@apache.org>.
liyubin117 commented on issue #5004:
URL: https://github.com/apache/paimon/issues/5004#issuecomment-2676869247

   @klyashko We have a table that has been delayed by the downstream task consumer, resulting in 500,000 files. After using `alter table set`, some files expire after each checkpoint, finally down to 5000 files.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org