You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/16 13:52:54 UTC
[GitHub] [iceberg] kunal-nandwana opened a new issue, #5548: Schema Evolution
kunal-nandwana opened a new issue, #5548:
URL: https://github.com/apache/iceberg/issues/5548
### Query engine
Spark
### Question
Hi Team.
Currently I am using Iceberg in my project, so I am having one double in that.
My Current Scenario:
1. I have loaded the data into my Iceberg table using spark data frame(this is my doing through spark job)
**df.writeTo("catalog.my.test2").using("iceberg").create()**
2. Now From source side I have added two colums and started the Job which is doing merge
**df.createOrReplaceTempView("myview")
spark.sql("MERGE INTO catalog.mydb.test2 as t USING (SELECT * FROM myview) as s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *")**
Doing both of these step I am expecting new columns to be added into the target table but it did not worked,
**As I can Iceberg Support full schema evolution.. What does it means..Which is not adding any columns dynamically to my target table.**
Please help how can I achieve adding new columns into my target table dynmically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #5548: Schema Evolution
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #5548:
URL: https://github.com/apache/iceberg/issues/5548#issuecomment-1218349861
I'm going to proactively close this issue as https://github.com/apache/iceberg/issues/5556 seems to have taken over it, but _please_ feel free to reopen if need be or you'd like to keep it open longer @kunal-nandwana. I'm not the arbiter of issues by any means, just trying to do some housekeeping 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick closed issue #5548: Schema Evolution
Posted by GitBox <gi...@apache.org>.
kbendick closed issue #5548: Schema Evolution
URL: https://github.com/apache/iceberg/issues/5548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #5548: Schema Evolution
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #5548:
URL: https://github.com/apache/iceberg/issues/5548#issuecomment-1218347483
May I close this issue then @kunal-nandwana (assuming that your question has been answered and the feature request links back to it if need be)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #5548: Schema Evolution
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #5548:
URL: https://github.com/apache/iceberg/issues/5548#issuecomment-1217099105
For the schema evolution, normally you should run the `ALTER TABLE ... ADD COLUMN` commands: https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table--add-column
If you want to have the schema updated during `merge into` without specifying it, you can use the `.option("mergeSchema", true)` when writing to the table.
However, I'm not sure if that option is possible to use from SQL at the moment. We might need to capture the comments and apply them as hints (which isn't supported at this time afaik): https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#hints
For writes that can use the dataframe writer (which doesn't include `MERGE INTO` unfortunately), you could use the `mergeSchema` option and get the behavior you want.
If you're interested in possibly getting the `mergeSchema` option as a hint, I would suggest opening a new feature request issue to request just that (linking to this one if you'd like). I'm not sure if it can be done, but as a feature request issue with a title like "Support Hints for Dataframe Writer Options Like 'mergeSchema'" and then explain your MERGE INTO case, that would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kunal-nandwana commented on issue #5548: Schema Evolution
Posted by GitBox <gi...@apache.org>.
kunal-nandwana commented on issue #5548:
URL: https://github.com/apache/iceberg/issues/5548#issuecomment-1217460389
@kbendick thanks for replying.. I have raised the new feature request.. I hope I get any positive response for that..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org