You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/01/18 17:37:27 UTC
[jira] [Updated] (HADOOP-13985) s3guard: add a version marker to
every table
[ https://issues.apache.org/jira/browse/HADOOP-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-13985:
------------------------------------
Attachment: HADOOP-13985-HADOOP-13345-001.patch
patch 001: version marker, with tests and documentation; tests cover marshall/unmarshall of marker; how the client reacts to incompatible versions.
The get of the marker replaces {{table.describe()}} as the initialization probe; it's then compared to the compile-time version. No marker: Exception. Version mismatch (currently) exception. If anyone does incompatibly update versions, they will get to handle the upgrade mechanism here.
I've also pulled out table creation into its own method, combining that with the wait for table completion to finish. We should need to call {{awaitTable()}}.
The docs cover: error messages and declare a versioning policy.
Also: LambdaTestUtils is upgraded to handle lambda expressions which return void more elegantly
Tested: s3a ireland, skipping the failing Test* tests noted in HADOOP-13995. I haven't run it with HADOOP-13589 applied, so not performed full integration tests with AWS. I'd like to get that patch in so I can do this *before* applying this patch
> s3guard: add a version marker to every table
> --------------------------------------------
>
> Key: HADOOP-13985
> URL: https://issues.apache.org/jira/browse/HADOOP-13985
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Attachments: HADOOP-13985-HADOOP-13345-001.patch
>
>
> This is something else we need before any preview: a way to identify a table version, so that if future versions change the table structure:
> * older clients can recognise that it's a newer format, and fail
> * the future version can identify that it's an older format, and fail until some fsck-upgrade operation has taken place
> I think something like a row on a path which is impossible in a real filesystem, such as "../VERSION" would allow a version marker to go in; the length field could be abused for the version number.
> This field would be something that'd be checked in init(), so be the simple test for table existence we need for faster init
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org