You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/02 10:57:36 UTC
[GitHub] [iceberg] AlvaroNavarroMora opened a new issue, #6105: Add tblproperties using alter table with spark
AlvaroNavarroMora opened a new issue, #6105:
URL: https://github.com/apache/iceberg/issues/6105
### Apache Iceberg version
0.13.1
### Query engine
Spark
### Please describe the bug 🐞
We are trying to add the following properties to existing iceberg tables using spark 3.1.2:
```
'write.metadata.delete-after-commit.enabled'='true'
'write.metadata.previous-versions-max'='10'
```
In order to do so we are using the next queries:
```
ALTER TABLE <tableName> SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled'='true')
ALTER TABLE <tableName> SET TBLPROPERTIES ('write.metadata.previous-versions-max'='10')
```
The queries are being executed without any errors, but after reviewing the tables the properties we observe that the properties have not changed.
We can't add the properties when creating the table because we are using liquibase with the athena connector, which does not support Iceberg specific properties. This is why we need to add the properties afterwards.
Is it possible to add these properties to existing tables using spark in any other way? if that is the case, how should we do it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] AlvaroNavarroMora commented on issue #6105: Add tblproperties using alter table with spark
Posted by GitBox <gi...@apache.org>.
AlvaroNavarroMora commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1319803494
Hi, after doing some testing we have observed that the properties are being applied to the tables, however it seems to not be working properly. We have tables with those properties applied that have over 4k metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6105: Add tblproperties using alter table with spark
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1595902412
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6105: Add tblproperties using alter table with spark
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1313529395
@AlvaroNavarroMora I don't see any reason why those table properties wouldn't be reflected/set. Are you running `SHOW TBLPROPERTIES <tblName>` or something else to describe the table properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6105: Add tblproperties using alter table with spark
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1553827040
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ELHoussineT commented on issue #6105: Add tblproperties using alter table with spark
Posted by GitBox <gi...@apache.org>.
ELHoussineT commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1319834985
@AlvaroNavarroMora Interesting, we did as the example you shared in the main thread and we saw that the metadata were deleted. For verification of the deletion of the metadata, we used pyiceberg from this PR https://github.com/apache/iceberg/pull/6034.
Running the following code before and after setting the table properties showed that the old metadata were in fact deleted.
```
from pyiceberg.catalog.glue import GlueCatalog
catalog = GlueCatalog("glue", warehouse=f"s3://...")
table = catalog.load_table("<glue db>.<table>")
print(table.metadata.metadata_log)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6105: Add tblproperties using alter table with spark
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6105: Add tblproperties using alter table with spark
URL: https://github.com/apache/iceberg/issues/6105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ELHoussineT commented on issue #6105: Add tblproperties using alter table with spark
Posted by GitBox <gi...@apache.org>.
ELHoussineT commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1318661985
@AlvaroNavarroMora Did it work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6105: Add tblproperties using alter table with spark
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6105:
URL: https://github.com/apache/iceberg/issues/6105#issuecomment-1319810330
The same issue came up in a [Slack discussion](https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1668714478198079?thread_ts=1668645354.671479&cid=C025PH0G1D4) and I've opened https://github.com/apache/iceberg/pull/6217 to address that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org