You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/10 09:15:52 UTC
[GitHub] [hudi] 0x574C opened a new issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
0x574C opened a new issue #3960:
URL: https://github.com/apache/hudi/issues/3960
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a mysql-cdc table A_cdc to capture mysql table A by flink sql
2. Create a hudi table B_hudi to write data to hive table B by flink sql
3. Execute flink sql `insert into B_hudi select * from A_cdc`
4. Add a column to table A
It seems that if you want to add columns, you must redefine the schema and restart the task.
How to sync "add column" event without restart?
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-1025282665
> IMO, this requires the flink can support schema evolution, current community flink doesn't support it, you can consider use aliyun enterprise flink, we are support it.
@0x574C i hope this gives you a good idea about supporting schema evolution in flink. If you don't have further questions, we may close this. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lsyldliu commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
lsyldliu commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-985950976
IMO, this requires the flink can support schema evolution, current community flink doesn't support it, you can consider use aliyun enterprise flink, we are support it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-965940585
@danny0405 is schema evolution supported with flink writer? i recall it isn't but not 100% sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-1067592270
@danny0405 : is there any follow up required on this github issue. if not, can we close it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-1055119077
> > IMO, this requires the flink can support schema evolution, current community flink doesn't support it, you can consider use aliyun enterprise flink, we are support it.
>
> @0x574C i hope this gives you a good idea about supporting schema evolution in flink. If you don't have further questions, we may close this. Thanks.
Yes, this feature is not supported yet mainly because there is no solution for adapter the schema changes of the internal row for flink engine, and for SQL semantics the row schema is solid, we may need some tricky syntax for that such as `create table target as select * from src_table` and the engine can handle the schema changes with varying SE/DE automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3960: [SUPPORT]How to auto sync "add column" by flink ?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3960:
URL: https://github.com/apache/hudi/issues/3960#issuecomment-1018602232
@danny0405 : can we have some follow up here please. If its a feature request, can we file a tracking jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org