You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2023/01/18 17:27:06 UTC
[GitHub] [iceberg] anthonysgro opened a new issue, #6619: Disaster Recovery Options for AWS Athena/Iceberg Integration
anthonysgro opened a new issue, #6619:
URL: https://github.com/apache/iceberg/issues/6619
### Query engine
AWS Athena
### Question
Right now, I have an Iceberg table set up through AWS Athena. I have backed up my data through AWS Backup, which basically restores the S3 buckets if for some reason the S3 buckets are deleted. I have tested this, and after restoring, Athena still lets me query the restored data, which is perfect.
If I run a DROP TABLE command, Athena loses reference to the table and the S3 data is deleted. I thought "okay, no problem, we can just recreate the table and then instantiate our backup". But this does not work. For some reason, Athena loses reference to the table even if I point it to the same exact location where it used to point. Does anyone have a solution for this? Right now, I am exposed to an erroneous DROP TABLE command by a developer/admin.
I have not been able to find any information on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on issue #6619: Disaster Recovery Options for AWS Athena/Iceberg Integration
Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-1397888851
Thanks for creating this issue, @anthonysgro could you provide more details on how you're recreating the table and pointing the location and how AWS Backup fits in?
As far as my understanding goes, for disaster recovery for Iceberg tables generally it's recommended to use S3 multi-region access points . You can check out https://github.com/apache/iceberg/issues/5779 this discussion for more details.
Some docs:
https://aws.amazon.com/s3/features/multi-region-access-points/
https://iceberg.apache.org/docs/latest/aws/#s3-access-points
If you desire S3 access point integration in Athena specifically, feel free to ping me on Slack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]
Posted by "a-agmon (via GitHub)" <gi...@apache.org>.
a-agmon commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-1975202872
@amogh-jahagirdar - I just wanted to say that we had a very problematic recovery case today, and your comment saved the day. We simply changed the metadata_location in glue and were able to recover :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on issue #6619: Disaster Recovery Options for AWS Athena/Iceberg Integration
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-1405811455
> Is there something with a metadata pointer or something if this occurs?
Yes, so Athena's integration with Iceberg uses the Glue Catalog. Every write done by Athena on the Iceberg table (in this case your inserts) is optimistically writing out the metadata file and then updating the metadata_location property on the Glue table as part of the commit.
When you drop an Iceberg table in Athena as you observed, we do delete the S3 files and drop the glue table (meaning now the pointer to the metadata location is lost). When you re-create the table after restoring the backup, a new glue table is created with a new metadata pointer (effectively an empty table state).
So after re-creating the table and restoring your data what you can do is update the metadata_location table property in Glue to point to the latest metadata location prior to dropping the table. After that you should be able to read the data as before. I was able to repro this process in my account, but please let me know if you run into any issues!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] anthonysgro commented on issue #6619: Disaster Recovery Options for AWS Athena/Iceberg Integration
Posted by "anthonysgro (via GitHub)" <gi...@apache.org>.
anthonysgro commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-1404528549
Yes. So here is specifically how it happens:
Creating my table:
I create my table through an Athena query
```
CREATE TABLE IF NOT EXISTS db.friends (
id string,
name string,
birthday timestamp
)
LOCATION s3://iceberg-datalake
TBLPROPERTIES (
'table_type' = 'ICEBERG'
)
```
We can insert a couple rows into this table, yada yada.
Then comes AWS Backup. It is a service that is natively integrated into S3. I went to the AWS Backup console, navigated to **Protected Resources**, and picked my iceberg datalake bucket, and created a recovery point. It takes about an hour or so for it to get backed up.
Once a recovery point becomes available, I deleted my entire s3://iceberg-datalake bucket. If I query my iceberg table through Athena, it throws an error (as expected, the data is deleted!). But no worries, I restored the bucket with a click of a button in AWS Backup, and voila, it recreates after a bit of time. I can once again query my iceberg table, as if nothing happened. Perfect.
However, some cases where this doesn't work:
If I run:
```
DROP TABLE db.friends
```
it deletes the table and the S3 data, as expected. AWS tells me iceberg tables do that, for some reason. But then, if I recreate the table using the above command, and then restore my backup, it throws a strange error, and it can't seem to get back to normal.
Is there something with a metadata pointer or something if this occurs? I haven't been able to find a way to restore iceberg table data after a DROP TABLE command. I would absolutely be interested in hearing more if this S3 access point integration helps prevent agains this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-2770956423
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]
Posted by "SamRaza356 (via GitHub)" <gi...@apache.org>.
SamRaza356 commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-2095687386
There is another issue the files present in metadata(.json & .avro) files are having original path locations that causes an issue if original table is dropped. So what we can do in that case, I think reading (.avro) files and changing path locations how we can do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6619: Disaster Recovery Options for AWS Athena/Iceberg Integration
URL: https://github.com/apache/iceberg/issues/6619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Disaster Recovery Options for AWS Athena/Iceberg Integration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6619:
URL: https://github.com/apache/iceberg/issues/6619#issuecomment-2807832660
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org