Schema Migration (example using Curator)

[David Medinets <da...@gmail.com>]

http://www.pgrs.net/2012/03/08/data-migrations-for-nosql-with-curator/

Would something like this work with Accumulo? Perhaps reserve a column
family ($SV$ - schema version) name? Maybe a bit of groovy (or other
dynamic language) can be attached to each table/schema version
combination so that the database holds all relevant information - even
the migration code.

Re: Schema Migration (example using Curator)

[Billie J Rinaldi <bi...@ugov.gov>]

On Friday, March 16, 2012 7:11:19 AM, "David Medinets" <da...@gmail.com> wrote:
> http://www.pgrs.net/2012/03/08/data-migrations-for-nosql-with-curator/
> 
> Would something like this work with Accumulo? Perhaps reserve a column
> family ($SV$ - schema version) name? Maybe a bit of groovy (or other
> dynamic language) can be attached to each table/schema version
> combination so that the database holds all relevant information - even
> the migration code.

In Accumulo you could use an Iterator for data migrations.  The iterator could be applied at compaction time and scan time so that any clients querying the table would see the new schema, and the schema change would lazily be written to disk as more data is written to the table and files are compacted.  In 1.3 it would be possible to tell the table to compact anything that hadn't been compacted since a certain date (e.g. the date the iterator was applied), so you could decide at some point to complete the migration and remove the Iterator.  This is no longer possible in 1.4 because you aren't allowed to request a compaction for a date, you are only allowed to request a compaction to be started immediately.  If you wanted to be able to remove the migration iterator, you would have to inspect the tablets manually to see if they'd been compacted since the migration began and request them to compact if they hadn't.

Perhaps we could create an ability to mark a table, and an ability to compact anything that hasn't been compacted since the mark?

Billie