Re: data extractor

[Nikolay Izhikov <ni...@apache.org>]

Hello, Dmitriy.

Should we support extraction of encrypted data?

There will be 2 type of keys we should load to successfully extract data:

* master key: keystore + password required.
* cache keys: masterkey + access to metastore required.

TDE task is almost done, please, take a look.

ticket - https://issues.apache.org/jira/browse/IGNITE-8485
prototype - https://github.com/apache/ignite/pull/4167
spi - https://github.com/apache/ignite/pull/4167/files#diff-9a792ab0e6971f202d22d530af0ac933

В Сб, 30/06/2018 в 22:37 +0300, Dmitriy Govorukhin пишет:
> Igniters,
> 
> I am working on IGNITE-7644 (export all key-value data from a persisted partition),
> it will be command line tool for extracting data from Ignite partition file without the need to start node.
> The main motivation is to have a lifebuoy in case if a file has damage for some reason. 
> 
> I suggest simple API and two commands for the first implementation:
> 
> -c
> --CRC [srcPath] - check CRC for all(or by type) pages in partition
> 
> -e
> --extract [srcPath] [outPath] - dump all survey data from partition to another file with raw key/value pair format 
> (required graceful stop for a node, not necessary after --restore will be implemented)
> 
> Output file format see in attached, this format does not contain any index inside but it is very simple and
> flexible for future works with raw key/value data.
> 
> Future features:
> -u
> --upload - reload raw key/value pairs to node
> 
> -s
> --status - check current node file status, need binary recovery or not (node crash on the middle of a checkpoint)
> 
> -r
> --restore - restore binary consistency (finish checkpoint, required WAL file for recovery)
> 
> Let's start a discussion, any comments are welcome.
> 

Re: data extractor

[Dmitriy Govorukhin <dm...@gmail.com>]

Nikolay,

I think we won't support extract from encrypted store In the first
implementation.
I guess we can support the encrypted store in future, or you have a reason
why we should do it in first?


On Sun, Jul 1, 2018 at 11:48 AM Nikolay Izhikov <ni...@apache.org> wrote:

> Hello, Dmitriy.
>
> Should we support extraction of encrypted data?
>
> There will be 2 type of keys we should load to successfully extract data:
>
> * master key: keystore + password required.
> * cache keys: masterkey + access to metastore required.
>
> TDE task is almost done, please, take a look.
>
> ticket - https://issues.apache.org/jira/browse/IGNITE-8485
> prototype - https://github.com/apache/ignite/pull/4167
> spi -
> https://github.com/apache/ignite/pull/4167/files#diff-9a792ab0e6971f202d22d530af0ac933
>
> В Сб, 30/06/2018 в 22:37 +0300, Dmitriy Govorukhin пишет:
> > Igniters,
> >
> > I am working on IGNITE-7644 (export all key-value data from a persisted
> partition),
> > it will be command line tool for extracting data from Ignite partition
> file without the need to start node.
> > The main motivation is to have a lifebuoy in case if a file has damage
> for some reason.
> >
> > I suggest simple API and two commands for the first implementation:
> >
> > -c
> > --CRC [srcPath] - check CRC for all(or by type) pages in partition
> >
> > -e
> > --extract [srcPath] [outPath] - dump all survey data from partition to
> another file with raw key/value pair format
> > (required graceful stop for a node, not necessary after --restore will
> be implemented)
> >
> > Output file format see in attached, this format does not contain any
> index inside but it is very simple and
> > flexible for future works with raw key/value data.
> >
> > Future features:
> > -u
> > --upload - reload raw key/value pairs to node
> >
> > -s
> > --status - check current node file status, need binary recovery or not
> (node crash on the middle of a checkpoint)
> >
> > -r
> > --restore - restore binary consistency (finish checkpoint, required WAL
> file for recovery)
> >
> > Let's start a discussion, any comments are welcome.
> >