You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@asterixdb.apache.org by Murtadha Hubail <hu...@gmail.com> on 2017/11/27 23:07:33 UTC
Storage Structure Change
Hi all,
Recently, a feature was added to AsterixDB to allow a dataset to be rebalanced from a set of nodes (a node group) to another node group. Rebalancing a dataset will result in redistributing the dataset’s data to the nodes in the new node group. Since a single node may appear in both the old and the new node group, a new level in the storage tree had to be introduced to distinguish between the old and the new copies of the dataset. To do that without breaking backward compatibility for existing users who would like to upgrade binaries without having to reload their existing data, we added a migration utility that will run on the node startup and migrate the data to the new storage structure. However, it is recommended that you backup your existing data to avoid any permanent data loss in case of any unexpected failures during the migration process.
Cheers,
Murtadha
Re: Storage Structure Change
Posted by Murtadha Hubail <hu...@gmail.com>.
The migration utility is built inside the system as part of nodes startup sequence and is available in the current master. When a storage version that requires migration is detected on the node startup, it will be automatically triggered.
As for the backup, taking a copy of each configured storage directory as well as the transaction logs directory in any format, compressed or uncompressed, should be sufficient. In case of any failure during the migration, using the old binaries with the backup copy should work.
Cheers,
Murtadha
On 11/29/2017, 12:44 AM, "Michael Carey" <mj...@ics.uci.edu> wrote:
Murtadha,
When the utility is ready, could you post another note on where it lives
and how to use it and also what the best practice advice is on how to
back up before proceeding? Thx!
Cheers,
Mike
On 11/27/17 3:07 PM, Murtadha Hubail wrote:
>
> Hi all,
>
> Recently, a feature was added to AsterixDB to allow a dataset to be
> rebalanced from a set of nodes (a node group) to another node group.
> Rebalancing a dataset will result in redistributing the dataset’s data
> to the nodes in the new node group. Since a single node may appear in
> both the old and the new node group, a new level in the storage tree
> had to be introduced to distinguish between the old and the new copies
> of the dataset. To do that without breaking backward compatibility for
> existing users who would like to upgrade binaries without having to
> reload their existing data, we added a migration utility that will run
> on the node startup and migrate the data to the new storage structure.
> *_However, it is recommended that you backup your existing data to
> avoid any permanent data loss in case of any unexpected failures
> during the migration process_*_._
>
> Cheers,
>
> Murtadha
>
Re: Storage Structure Change
Posted by Murtadha Hubail <hu...@gmail.com>.
The migration utility is built inside the system as part of nodes startup sequence and is available in the current master. When a storage version that requires migration is detected on the node startup, it will be automatically triggered.
As for the backup, taking a copy of each configured storage directory as well as the transaction logs directory in any format, compressed or uncompressed, should be sufficient. In case of any failure during the migration, using the old binaries with the backup copy should work.
Cheers,
Murtadha
On 11/29/2017, 12:44 AM, "Michael Carey" <mj...@ics.uci.edu> wrote:
Murtadha,
When the utility is ready, could you post another note on where it lives
and how to use it and also what the best practice advice is on how to
back up before proceeding? Thx!
Cheers,
Mike
On 11/27/17 3:07 PM, Murtadha Hubail wrote:
>
> Hi all,
>
> Recently, a feature was added to AsterixDB to allow a dataset to be
> rebalanced from a set of nodes (a node group) to another node group.
> Rebalancing a dataset will result in redistributing the dataset’s data
> to the nodes in the new node group. Since a single node may appear in
> both the old and the new node group, a new level in the storage tree
> had to be introduced to distinguish between the old and the new copies
> of the dataset. To do that without breaking backward compatibility for
> existing users who would like to upgrade binaries without having to
> reload their existing data, we added a migration utility that will run
> on the node startup and migrate the data to the new storage structure.
> *_However, it is recommended that you backup your existing data to
> avoid any permanent data loss in case of any unexpected failures
> during the migration process_*_._
>
> Cheers,
>
> Murtadha
>
Re: Storage Structure Change
Posted by Michael Carey <mj...@ics.uci.edu>.
Murtadha,
When the utility is ready, could you post another note on where it lives
and how to use it and also what the best practice advice is on how to
back up before proceeding? Thx!
Cheers,
Mike
On 11/27/17 3:07 PM, Murtadha Hubail wrote:
>
> Hi all,
>
> Recently, a feature was added to AsterixDB to allow a dataset to be
> rebalanced from a set of nodes (a node group) to another node group.
> Rebalancing a dataset will result in redistributing the dataset’s data
> to the nodes in the new node group. Since a single node may appear in
> both the old and the new node group, a new level in the storage tree
> had to be introduced to distinguish between the old and the new copies
> of the dataset. To do that without breaking backward compatibility for
> existing users who would like to upgrade binaries without having to
> reload their existing data, we added a migration utility that will run
> on the node startup and migrate the data to the new storage structure.
> *_However, it is recommended that you backup your existing data to
> avoid any permanent data loss in case of any unexpected failures
> during the migration process_*_._
>
> Cheers,
>
> Murtadha
>
Re: Storage Structure Change
Posted by Michael Carey <mj...@ics.uci.edu>.
Murtadha,
When the utility is ready, could you post another note on where it lives
and how to use it and also what the best practice advice is on how to
back up before proceeding? Thx!
Cheers,
Mike
On 11/27/17 3:07 PM, Murtadha Hubail wrote:
>
> Hi all,
>
> Recently, a feature was added to AsterixDB to allow a dataset to be
> rebalanced from a set of nodes (a node group) to another node group.
> Rebalancing a dataset will result in redistributing the dataset’s data
> to the nodes in the new node group. Since a single node may appear in
> both the old and the new node group, a new level in the storage tree
> had to be introduced to distinguish between the old and the new copies
> of the dataset. To do that without breaking backward compatibility for
> existing users who would like to upgrade binaries without having to
> reload their existing data, we added a migration utility that will run
> on the node startup and migrate the data to the new storage structure.
> *_However, it is recommended that you backup your existing data to
> avoid any permanent data loss in case of any unexpected failures
> during the migration process_*_._
>
> Cheers,
>
> Murtadha
>