You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/01 17:17:14 UTC
[GitHub] [iceberg] alesk opened a new issue, #6340: Deleting column from an iceberg table breaks schema in AWS Glue catalog
alesk opened a new issue, #6340:
URL: https://github.com/apache/iceberg/issues/6340
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When deleting a column with Spark SQL using `ALTER TABLE table_name DROP COLUMN column_name`, the following happens:
1) The column gets removed from the current schema in the Table's metadata on s3.
2) A new version of the schema is created in AWS Glue but the column is moved to the last position in the table instead of being removed.
The two schemas have diverged. Querying with Spark works as expected but using Athena the wrong schema is reported thus breaking tools such as Metabase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Deleting a column from an iceberg table breaks schema in AWS Glue catalog [iceberg]
Posted by "qoqajr (via GitHub)" <gi...@apache.org>.
qoqajr commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1867417317
I did the same but having to write custom code for both deleting a column with spark.sql and then in the glue catalog (I did it with awswrangler lib) is a bit messy to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1592149943
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] mattiamatrix commented on issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by "mattiamatrix (via GitHub)" <gi...@apache.org>.
mattiamatrix commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1625250613
Hello, any updates on this? This bug is still present!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1334099047
@amogh-jahagirdar could you please take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Deleting a column from an iceberg table breaks schema in AWS Glue catalog [iceberg]
Posted by "qoqajr (via GitHub)" <gi...@apache.org>.
qoqajr commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1866356484
Hello, any updates on this? This bug is still present!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
URL: https://github.com/apache/iceberg/issues/6340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1569307699
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6340: Deleting a column from an iceberg table breaks schema in AWS Glue catalog
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1625330855
@mattiamatrix since this seems to be an Athena-specific issue, it's probably best to take this issue to the Athena team
/cc @jackye1995 @amogh-jahagirdar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Deleting a column from an iceberg table breaks schema in AWS Glue catalog [iceberg]
Posted by "00Fede (via GitHub)" <gi...@apache.org>.
00Fede commented on issue #6340:
URL: https://github.com/apache/iceberg/issues/6340#issuecomment-1867021274
This is still happening. We have to delete the dropped column directly from AWS Glue Data Catalog to be able to run queries in Athena or Quicksight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org