You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Ted Yu <yu...@gmail.com> on 2013/04/16 04:13:32 UTC
collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Hi,
This afternoon Huned ad I discovered an issue while playing with HBase
Snapshots on top of Hadoop's Snapshot branch (
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
component while HBase tries to create paths with .snapshot as a component.
This leads to issues in HBase, and one of HDFS or HBase needs to give up
the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
(quite recently) and it may not be too late to change HBase to use a
different path component in an upcoming new release.
In HBase these path names are not user visible. If there is a deployment of
0.94.6, one could provide a migration tool that renames .snapshot to
.hbase-snapshot or something to be able to move to the Snapshot release of
Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
a convention that is used by many file systems. It's a matter of
familiarity with such path names that would help users in using HDFS
snapshots.
I am including the hdfs-dev in this email. Would appreciate if we could
work together and come up with a solution.
You can find sample output from hdfs command here:
http://pastebin.com/bBqR4Fvr
Cheers
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Putting back dev@hbase.
".hbase-sanpshot" would be created at cluster startup. After that user
wouldn't be able to use the same directory name.
On Mon, Apr 15, 2013 at 8:23 PM, Azuryy Yu <az...@gmail.com> wrote:
> I think ".hbase-sanpshot" is good, but we should also disallow user to
> create ".hbase-sanpshot" under hbase.root sub directories.
>
>
> On Tue, Apr 16, 2013 at 11:18 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot"
> >
> > Please suggest better name in future correspondence.
> >
> > Thanks
> >
> > On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Lars:
> > > I will go ahead and log an HBase JIRA, tomorrow morning.
> > >
> > > This hopefully would give people enough time to respond.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >
> > >> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> > >>
> > >> -- Lars
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Ted Yu <yu...@gmail.com>
> > >> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > >> Sent: Monday, April 15, 2013 7:13 PM
> > >> Subject: collision in the naming of '.snapshot' directory between hdfs
> > >> snapshot and hbase snapshot
> > >>
> > >>
> > >> Hi,
> > >> This afternoon Huned ad I discovered an issue while playing with HBase
> > >> Snapshots on top of Hadoop's Snapshot branch (
> > >> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > >>
> > >> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> > as a
> > >> component while HBase tries to create paths with .snapshot as a
> > component.
> > >> This leads to issues in HBase, and one of HDFS or HBase needs to give
> up
> > >> the .snapshot reserved keyword. HBase released Snapshots feature in
> > 0.94.6
> > >> (quite recently) and it may not be too late to change HBase to use a
> > >> different path component in an upcoming new release.
> > >>
> > >> In HBase these path names are not user visible. If there is a
> deployment
> > >> of
> > >> 0.94.6, one could provide a migration tool that renames .snapshot to
> > >> .hbase-snapshot or something to be able to move to the Snapshot
> release
> > of
> > >> Hadoop. On the other hand, .snapshot in HDFS is a user visible name
> and
> > is
> > >> a convention that is used by many file systems. It's a matter of
> > >> familiarity with such path names that would help users in using HDFS
> > >> snapshots.
> > >>
> > >> I am including the hdfs-dev in this email. Would appreciate if we
> could
> > >> work together and come up with a solution.
> > >>
> > >> You can find sample output from hdfs command here:
> > >> http://pastebin.com/bBqR4Fvr
> > >>
> > >> Cheers
> > >>
> > >
> > >
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Putting back dev@hbase.
".hbase-sanpshot" would be created at cluster startup. After that user
wouldn't be able to use the same directory name.
On Mon, Apr 15, 2013 at 8:23 PM, Azuryy Yu <az...@gmail.com> wrote:
> I think ".hbase-sanpshot" is good, but we should also disallow user to
> create ".hbase-sanpshot" under hbase.root sub directories.
>
>
> On Tue, Apr 16, 2013 at 11:18 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot"
> >
> > Please suggest better name in future correspondence.
> >
> > Thanks
> >
> > On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Lars:
> > > I will go ahead and log an HBase JIRA, tomorrow morning.
> > >
> > > This hopefully would give people enough time to respond.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >
> > >> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> > >>
> > >> -- Lars
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Ted Yu <yu...@gmail.com>
> > >> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > >> Sent: Monday, April 15, 2013 7:13 PM
> > >> Subject: collision in the naming of '.snapshot' directory between hdfs
> > >> snapshot and hbase snapshot
> > >>
> > >>
> > >> Hi,
> > >> This afternoon Huned ad I discovered an issue while playing with HBase
> > >> Snapshots on top of Hadoop's Snapshot branch (
> > >> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > >>
> > >> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> > as a
> > >> component while HBase tries to create paths with .snapshot as a
> > component.
> > >> This leads to issues in HBase, and one of HDFS or HBase needs to give
> up
> > >> the .snapshot reserved keyword. HBase released Snapshots feature in
> > 0.94.6
> > >> (quite recently) and it may not be too late to change HBase to use a
> > >> different path component in an upcoming new release.
> > >>
> > >> In HBase these path names are not user visible. If there is a
> deployment
> > >> of
> > >> 0.94.6, one could provide a migration tool that renames .snapshot to
> > >> .hbase-snapshot or something to be able to move to the Snapshot
> release
> > of
> > >> Hadoop. On the other hand, .snapshot in HDFS is a user visible name
> and
> > is
> > >> a convention that is used by many file systems. It's a matter of
> > >> familiarity with such path names that would help users in using HDFS
> > >> snapshots.
> > >>
> > >> I am including the hdfs-dev in this email. Would appreciate if we
> could
> > >> work together and come up with a solution.
> > >>
> > >> You can find sample output from hdfs command here:
> > >> http://pastebin.com/bBqR4Fvr
> > >>
> > >> Cheers
> > >>
> > >
> > >
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Azuryy Yu <az...@gmail.com>.
I think ".hbase-sanpshot" is good, but we should also disallow user to
create ".hbase-sanpshot" under hbase.root sub directories.
On Tue, Apr 16, 2013 at 11:18 AM, Ted Yu <yu...@gmail.com> wrote:
> I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot"
>
> Please suggest better name in future correspondence.
>
> Thanks
>
> On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Lars:
> > I will go ahead and log an HBase JIRA, tomorrow morning.
> >
> > This hopefully would give people enough time to respond.
> >
> > Cheers
> >
> >
> > On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org> wrote:
> >
> >> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Ted Yu <yu...@gmail.com>
> >> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> >> Sent: Monday, April 15, 2013 7:13 PM
> >> Subject: collision in the naming of '.snapshot' directory between hdfs
> >> snapshot and hbase snapshot
> >>
> >>
> >> Hi,
> >> This afternoon Huned ad I discovered an issue while playing with HBase
> >> Snapshots on top of Hadoop's Snapshot branch (
> >> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> >>
> >> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> as a
> >> component while HBase tries to create paths with .snapshot as a
> component.
> >> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> >> the .snapshot reserved keyword. HBase released Snapshots feature in
> 0.94.6
> >> (quite recently) and it may not be too late to change HBase to use a
> >> different path component in an upcoming new release.
> >>
> >> In HBase these path names are not user visible. If there is a deployment
> >> of
> >> 0.94.6, one could provide a migration tool that renames .snapshot to
> >> .hbase-snapshot or something to be able to move to the Snapshot release
> of
> >> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and
> is
> >> a convention that is used by many file systems. It's a matter of
> >> familiarity with such path names that would help users in using HDFS
> >> snapshots.
> >>
> >> I am including the hdfs-dev in this email. Would appreciate if we could
> >> work together and come up with a solution.
> >>
> >> You can find sample output from hdfs command here:
> >> http://pastebin.com/bBqR4Fvr
> >>
> >> Cheers
> >>
> >
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot"
Please suggest better name in future correspondence.
Thanks
On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <yu...@gmail.com> wrote:
> Lars:
> I will go ahead and log an HBase JIRA, tomorrow morning.
>
> This hopefully would give people enough time to respond.
>
> Cheers
>
>
> On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org> wrote:
>
>> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> Sent: Monday, April 15, 2013 7:13 PM
>> Subject: collision in the naming of '.snapshot' directory between hdfs
>> snapshot and hbase snapshot
>>
>>
>> Hi,
>> This afternoon Huned ad I discovered an issue while playing with HBase
>> Snapshots on top of Hadoop's Snapshot branch (
>> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>>
>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
>> component while HBase tries to create paths with .snapshot as a component.
>> This leads to issues in HBase, and one of HDFS or HBase needs to give up
>> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
>> (quite recently) and it may not be too late to change HBase to use a
>> different path component in an upcoming new release.
>>
>> In HBase these path names are not user visible. If there is a deployment
>> of
>> 0.94.6, one could provide a migration tool that renames .snapshot to
>> .hbase-snapshot or something to be able to move to the Snapshot release of
>> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
>> a convention that is used by many file systems. It's a matter of
>> familiarity with such path names that would help users in using HDFS
>> snapshots.
>>
>> I am including the hdfs-dev in this email. Would appreciate if we could
>> work together and come up with a solution.
>>
>> You can find sample output from hdfs command here:
>> http://pastebin.com/bBqR4Fvr
>>
>> Cheers
>>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot"
Please suggest better name in future correspondence.
Thanks
On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <yu...@gmail.com> wrote:
> Lars:
> I will go ahead and log an HBase JIRA, tomorrow morning.
>
> This hopefully would give people enough time to respond.
>
> Cheers
>
>
> On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org> wrote:
>
>> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> Sent: Monday, April 15, 2013 7:13 PM
>> Subject: collision in the naming of '.snapshot' directory between hdfs
>> snapshot and hbase snapshot
>>
>>
>> Hi,
>> This afternoon Huned ad I discovered an issue while playing with HBase
>> Snapshots on top of Hadoop's Snapshot branch (
>> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>>
>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
>> component while HBase tries to create paths with .snapshot as a component.
>> This leads to issues in HBase, and one of HDFS or HBase needs to give up
>> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
>> (quite recently) and it may not be too late to change HBase to use a
>> different path component in an upcoming new release.
>>
>> In HBase these path names are not user visible. If there is a deployment
>> of
>> 0.94.6, one could provide a migration tool that renames .snapshot to
>> .hbase-snapshot or something to be able to move to the Snapshot release of
>> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
>> a convention that is used by many file systems. It's a matter of
>> familiarity with such path names that would help users in using HDFS
>> snapshots.
>>
>> I am including the hdfs-dev in this email. Would appreciate if we could
>> work together and come up with a solution.
>>
>> You can find sample output from hdfs command here:
>> http://pastebin.com/bBqR4Fvr
>>
>> Cheers
>>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Lars:
I will go ahead and log an HBase JIRA, tomorrow morning.
This hopefully would give people enough time to respond.
Cheers
On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org> wrote:
> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>
> -- Lars
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
We can simplify this by just changing the snapshot dir name to .hbase_snapshot and publishing instructions about to rename the directory in case anybody used that feature in 0.94.6.
-- Lars
________________________________
From: Jonathan Hsieh <jo...@cloudera.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
Sent: Monday, April 15, 2013 8:24 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
constraints:
1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever
make it to 1.x. This hdfs feature ideally this shouldn't affect current A
pache Hbase 0.94.x's.
3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
pick a different table snapshot name to respect fs conventions.
proposed actions:
1) let's make the hbase snapshot for a conf variable. (hbase.
snapshots.dir) let's change the default for hbase 95+. (maybe
.hbase-snapshots). we'll also port this patch to 0.94.x
2) let's publish instructions on how to update the hbase snapshot dir:
shutdown hbase, config update, rename dir, restart hbase.
3) I lean towards leaving the current default hbase snapshot dir in 94
since it shouldn't be affected. upgrading hbase to 95/96 will require
shutdown and update scripts so it seems like the ideal time to autoforce
this default change.
Thoughts?
On Monday, April 15, 2013, lars hofhansl wrote:
> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>
> -- Lars
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I have logged HBASE-8352
Cheers
On Mon, Apr 15, 2013 at 9:17 PM, Ted Yu <yu...@gmail.com> wrote:
> I have a patch which touched these tests:
>
> http://pastebin.com/P4p8LEAZ
>
> I am running 0.94 test suite now - will publish patch and test result in
> the morning.
>
> Cheers
>
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. Alternatively, we can detect the underlying Hadoop version, and use
>> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>>
>> I think this would introduce more confusion, especially for operations.
>>
>> Cheers
>>
>> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>wrote:
>>
>>> Because HDFS exposes the snapshots so that the normal file system
>>> operations are mapped inside snapshot dirs, I think HDFS reserving the
>>> .snapshot name makes sense. OTOH, nothing is specific about the dir name
>>> that is chosen by HBase.
>>>
>>> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>>> being run on top of hadoop 2. Alternatively, we can detect the underlying
>>> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>>> depending on h1 & h2.
>>>
>>> Enis
>>>
>>>
>>> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> > bq. let's make the hbase snapshot for a conf variable.
>>> >
>>> > Once we decide on the new name of snapshot directory, we should still
>>> use
>>> > hardcoded value. This aligns with current code base:
>>> > See this snippet from HConstants:
>>> >
>>> > public static final List<String> HBASE_NON_TABLE_DIRS =
>>> >
>>> > Collections.unmodifiableList(Arrays.asList(new String[] {
>>> > HREGION_LOGDIR_NAME,
>>> >
>>> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>>> >
>>> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>>> SNAPSHOT_DIR_NAME,
>>> > HBASE_TEMP_DIRECTORY }));
>>> > Cheers
>>> >
>>> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>>> wrote:
>>> >
>>> > > constraints:
>>> > >
>>> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>>> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>>> will
>>> > ever
>>> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>>> current
>>> > A
>>> > > pache Hbase 0.94.x's.
>>> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>>> should
>>> > > pick a different table snapshot name to respect fs conventions.
>>> > >
>>> > > proposed actions:
>>> > >
>>> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>>> > > snapshots.dir) let's change the default for hbase 95+. (maybe
>>> > > .hbase-snapshots). we'll also port this patch to 0.94.x
>>> > > 2) let's publish instructions on how to update the hbase snapshot
>>> dir:
>>> > > shutdown hbase, config update, rename dir, restart hbase.
>>> > > 3) I lean towards leaving the current default hbase snapshot dir in
>>> 94
>>> > > since it shouldn't be affected. upgrading hbase to 95/96 will
>>> require
>>> > > shutdown and update scripts so it seems like the ideal time to
>>> autoforce
>>> > > this default change.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > >
>>> > > On Monday, April 15, 2013, lars hofhansl wrote:
>>> > >
>>> > > > OK. Let's try to fix that quickly, so that I can release HBase
>>> 0.94.7.
>>> > > >
>>> > > > -- Lars
>>> > > >
>>> > > >
>>> > > >
>>> > > > ________________________________
>>> > > > From: Ted Yu <yu...@gmail.com>
>>> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>>> > > > Sent: Monday, April 15, 2013 7:13 PM
>>> > > > Subject: collision in the naming of '.snapshot' directory between
>>> hdfs
>>> > > > snapshot and hbase snapshot
>>> > > >
>>> > > >
>>> > > > Hi,
>>> > > > This afternoon Huned ad I discovered an issue while playing with
>>> HBase
>>> > > > Snapshots on top of Hadoop's Snapshot branch (
>>> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>>> > > >
>>> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>>> .snapshot
>>> > as
>>> > > a
>>> > > > component while HBase tries to create paths with .snapshot as a
>>> > > component.
>>> > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>>> give
>>> > up
>>> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>>> > > 0.94.6
>>> > > > (quite recently) and it may not be too late to change HBase to use
>>> a
>>> > > > different path component in an upcoming new release.
>>> > > >
>>> > > > In HBase these path names are not user visible. If there is a
>>> > deployment
>>> > > of
>>> > > > 0.94.6, one could provide a migration tool that renames .snapshot
>>> to
>>> > > > .hbase-snapshot or something to be able to move to the Snapshot
>>> release
>>> > > of
>>> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
>>> name and
>>> > > is
>>> > > > a convention that is used by many file systems. It's a matter of
>>> > > > familiarity with such path names that would help users in using
>>> HDFS
>>> > > > snapshots.
>>> > > >
>>> > > > I am including the hdfs-dev in this email. Would appreciate if we
>>> could
>>> > > > work together and come up with a solution.
>>> > > >
>>> > > > You can find sample output from hdfs command here:
>>> > > > http://pastebin.com/bBqR4Fvr
>>> > > >
>>> > > > Cheers
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > // Jonathan Hsieh (shay)
>>> > > // Software Engineer, Cloudera
>>> > > // jon@cloudera.com
>>> > >
>>> >
>>>
>>
>>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I have logged HBASE-8352
Cheers
On Mon, Apr 15, 2013 at 9:17 PM, Ted Yu <yu...@gmail.com> wrote:
> I have a patch which touched these tests:
>
> http://pastebin.com/P4p8LEAZ
>
> I am running 0.94 test suite now - will publish patch and test result in
> the morning.
>
> Cheers
>
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. Alternatively, we can detect the underlying Hadoop version, and use
>> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>>
>> I think this would introduce more confusion, especially for operations.
>>
>> Cheers
>>
>> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>wrote:
>>
>>> Because HDFS exposes the snapshots so that the normal file system
>>> operations are mapped inside snapshot dirs, I think HDFS reserving the
>>> .snapshot name makes sense. OTOH, nothing is specific about the dir name
>>> that is chosen by HBase.
>>>
>>> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>>> being run on top of hadoop 2. Alternatively, we can detect the underlying
>>> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>>> depending on h1 & h2.
>>>
>>> Enis
>>>
>>>
>>> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> > bq. let's make the hbase snapshot for a conf variable.
>>> >
>>> > Once we decide on the new name of snapshot directory, we should still
>>> use
>>> > hardcoded value. This aligns with current code base:
>>> > See this snippet from HConstants:
>>> >
>>> > public static final List<String> HBASE_NON_TABLE_DIRS =
>>> >
>>> > Collections.unmodifiableList(Arrays.asList(new String[] {
>>> > HREGION_LOGDIR_NAME,
>>> >
>>> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>>> >
>>> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>>> SNAPSHOT_DIR_NAME,
>>> > HBASE_TEMP_DIRECTORY }));
>>> > Cheers
>>> >
>>> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>>> wrote:
>>> >
>>> > > constraints:
>>> > >
>>> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>>> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>>> will
>>> > ever
>>> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>>> current
>>> > A
>>> > > pache Hbase 0.94.x's.
>>> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>>> should
>>> > > pick a different table snapshot name to respect fs conventions.
>>> > >
>>> > > proposed actions:
>>> > >
>>> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>>> > > snapshots.dir) let's change the default for hbase 95+. (maybe
>>> > > .hbase-snapshots). we'll also port this patch to 0.94.x
>>> > > 2) let's publish instructions on how to update the hbase snapshot
>>> dir:
>>> > > shutdown hbase, config update, rename dir, restart hbase.
>>> > > 3) I lean towards leaving the current default hbase snapshot dir in
>>> 94
>>> > > since it shouldn't be affected. upgrading hbase to 95/96 will
>>> require
>>> > > shutdown and update scripts so it seems like the ideal time to
>>> autoforce
>>> > > this default change.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > >
>>> > > On Monday, April 15, 2013, lars hofhansl wrote:
>>> > >
>>> > > > OK. Let's try to fix that quickly, so that I can release HBase
>>> 0.94.7.
>>> > > >
>>> > > > -- Lars
>>> > > >
>>> > > >
>>> > > >
>>> > > > ________________________________
>>> > > > From: Ted Yu <yu...@gmail.com>
>>> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>>> > > > Sent: Monday, April 15, 2013 7:13 PM
>>> > > > Subject: collision in the naming of '.snapshot' directory between
>>> hdfs
>>> > > > snapshot and hbase snapshot
>>> > > >
>>> > > >
>>> > > > Hi,
>>> > > > This afternoon Huned ad I discovered an issue while playing with
>>> HBase
>>> > > > Snapshots on top of Hadoop's Snapshot branch (
>>> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>>> > > >
>>> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>>> .snapshot
>>> > as
>>> > > a
>>> > > > component while HBase tries to create paths with .snapshot as a
>>> > > component.
>>> > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>>> give
>>> > up
>>> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>>> > > 0.94.6
>>> > > > (quite recently) and it may not be too late to change HBase to use
>>> a
>>> > > > different path component in an upcoming new release.
>>> > > >
>>> > > > In HBase these path names are not user visible. If there is a
>>> > deployment
>>> > > of
>>> > > > 0.94.6, one could provide a migration tool that renames .snapshot
>>> to
>>> > > > .hbase-snapshot or something to be able to move to the Snapshot
>>> release
>>> > > of
>>> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
>>> name and
>>> > > is
>>> > > > a convention that is used by many file systems. It's a matter of
>>> > > > familiarity with such path names that would help users in using
>>> HDFS
>>> > > > snapshots.
>>> > > >
>>> > > > I am including the hdfs-dev in this email. Would appreciate if we
>>> could
>>> > > > work together and come up with a solution.
>>> > > >
>>> > > > You can find sample output from hdfs command here:
>>> > > > http://pastebin.com/bBqR4Fvr
>>> > > >
>>> > > > Cheers
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > // Jonathan Hsieh (shay)
>>> > > // Software Engineer, Cloudera
>>> > > // jon@cloudera.com
>>> > >
>>> >
>>>
>>
>>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I have a patch which touched these tests:
http://pastebin.com/P4p8LEAZ
I am running 0.94 test suite now - will publish patch and test result in
the morning.
Cheers
On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Alternatively, we can detect the underlying Hadoop version, and use
> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>
> I think this would introduce more confusion, especially for operations.
>
> Cheers
>
> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
>
>> Because HDFS exposes the snapshots so that the normal file system
>> operations are mapped inside snapshot dirs, I think HDFS reserving the
>> .snapshot name makes sense. OTOH, nothing is specific about the dir name
>> that is chosen by HBase.
>>
>> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>> being run on top of hadoop 2. Alternatively, we can detect the underlying
>> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> depending on h1 & h2.
>>
>> Enis
>>
>>
>> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > bq. let's make the hbase snapshot for a conf variable.
>> >
>> > Once we decide on the new name of snapshot directory, we should still
>> use
>> > hardcoded value. This aligns with current code base:
>> > See this snippet from HConstants:
>> >
>> > public static final List<String> HBASE_NON_TABLE_DIRS =
>> >
>> > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > HREGION_LOGDIR_NAME,
>> >
>> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> >
>> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
>> > HBASE_TEMP_DIRECTORY }));
>> > Cheers
>> >
>> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> wrote:
>> >
>> > > constraints:
>> > >
>> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
>> > ever
>> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> current
>> > A
>> > > pache Hbase 0.94.x's.
>> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> should
>> > > pick a different table snapshot name to respect fs conventions.
>> > >
>> > > proposed actions:
>> > >
>> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > 2) let's publish instructions on how to update the hbase snapshot dir:
>> > > shutdown hbase, config update, rename dir, restart hbase.
>> > > 3) I lean towards leaving the current default hbase snapshot dir in 94
>> > > since it shouldn't be affected. upgrading hbase to 95/96 will require
>> > > shutdown and update scripts so it seems like the ideal time to
>> autoforce
>> > > this default change.
>> > >
>> > > Thoughts?
>> > >
>> > >
>> > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > >
>> > > > OK. Let's try to fix that quickly, so that I can release HBase
>> 0.94.7.
>> > > >
>> > > > -- Lars
>> > > >
>> > > >
>> > > >
>> > > > ________________________________
>> > > > From: Ted Yu <yu...@gmail.com>
>> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > Subject: collision in the naming of '.snapshot' directory between
>> hdfs
>> > > > snapshot and hbase snapshot
>> > > >
>> > > >
>> > > > Hi,
>> > > > This afternoon Huned ad I discovered an issue while playing with
>> HBase
>> > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > >
>> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> .snapshot
>> > as
>> > > a
>> > > > component while HBase tries to create paths with .snapshot as a
>> > > component.
>> > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> give
>> > up
>> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>> > > 0.94.6
>> > > > (quite recently) and it may not be too late to change HBase to use a
>> > > > different path component in an upcoming new release.
>> > > >
>> > > > In HBase these path names are not user visible. If there is a
>> > deployment
>> > > of
>> > > > 0.94.6, one could provide a migration tool that renames .snapshot to
>> > > > .hbase-snapshot or something to be able to move to the Snapshot
>> release
>> > > of
>> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
>> and
>> > > is
>> > > > a convention that is used by many file systems. It's a matter of
>> > > > familiarity with such path names that would help users in using HDFS
>> > > > snapshots.
>> > > >
>> > > > I am including the hdfs-dev in this email. Would appreciate if we
>> could
>> > > > work together and come up with a solution.
>> > > >
>> > > > You can find sample output from hdfs command here:
>> > > > http://pastebin.com/bBqR4Fvr
>> > > >
>> > > > Cheers
>> > >
>> > >
>> > >
>> > > --
>> > > // Jonathan Hsieh (shay)
>> > > // Software Engineer, Cloudera
>> > > // jon@cloudera.com
>> > >
>> >
>>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Azuryy Yu <az...@gmail.com>.
good suggestion.
On Wed, Apr 17, 2013 at 4:10 PM, Harsh J <ha...@cloudera.com> wrote:
> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >> ------------------------------
> >> *From:* Ted Yu <yu...@gmail.com>
> >> *To:* dev@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
> feature.
> >> (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route, I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >> > > >
> >> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
> >> > > >
> >> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> >> > > > HREGION_LOGDIR_NAME,
> >> > > >
> >> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> >> > > >
> >> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> >> > SNAPSHOT_DIR_NAME,
> >> > > > HBASE_TEMP_DIRECTORY }));
> >> > > > Cheers
> >> > > >
> >> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jon@cloudera.com
> >
> >> > > wrote:
> >> > > >
> >> > > > > constraints:
> >> > > > >
> >> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> >> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that
> it
> >> > will
> >> > > > ever
> >> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> >> > > current
> >> > > > A
> >> > > > > pache Hbase 0.94.x's.
> >> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these
> versions
> >> > > should
> >> > > > > pick a different table snapshot name to respect fs conventions.
> >> > > > >
> >> > > > > proposed actions:
> >> > > > >
> >> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> >> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> >> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> >> > > > > 2) let's publish instructions on how to update the hbase
> snapshot
> >> > dir:
> >> > > > > shutdown hbase, config update, rename dir, restart hbase.
> >> > > > > 3) I lean towards leaving the current default hbase snapshot
> dir in
> >> > 94
> >> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> >> > require
> >> > > > > shutdown and update scripts so it seems like the ideal time to
> >> > > autoforce
> >> > > > > this default change.
> >> > > > >
> >> > > > > Thoughts?
> >> > > > >
> >> > > > >
> >> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
> >> > > > >
> >> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
> >> > > 0.94.7.
> >> > > > > >
> >> > > > > > -- Lars
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > ________________________________
> >> > > > > > From: Ted Yu <yu...@gmail.com>
> >> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> >> > > > > > Sent: Monday, April 15, 2013 7:13 PM
> >> > > > > > Subject: collision in the naming of '.snapshot' directory
> between
> >> > > hdfs
> >> > > > > > snapshot and hbase snapshot
> >> > > > > >
> >> > > > > >
> >> > > > > > Hi,
> >> > > > > > This afternoon Huned ad I discovered an issue while playing
> with
> >> > > HBase
> >> > > > > > Snapshots on top of Hadoop's Snapshot branch (
> >> > > > > >
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> >> > > > > >
> >> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> >> > .snapshot
> >> > > > as
> >> > > > > a
> >> > > > > > component while HBase tries to create paths with .snapshot as
> a
> >> > > > > component.
> >> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs
> to
> >> > give
> >> > > > up
> >> > > > > > the .snapshot reserved keyword. HBase released Snapshots
> feature
> >> in
> >> > > > > 0.94.6
> >> > > > > > (quite recently) and it may not be too late to change HBase to
> >> use
> >> > a
> >> > > > > > different path component in an upcoming new release.
> >> > > > > >
> >> > > > > > In HBase these path names are not user visible. If there is a
> >> > > > deployment
> >> > > > > of
> >> > > > > > 0.94.6, one could provide a migration tool that renames
> .snapshot
> >> > to
> >> > > > > > .hbase-snapshot or something to be able to move to the
> Snapshot
> >> > > release
> >> > > > > of
> >> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
> >> name
> >> > > and
> >> > > > > is
> >> > > > > > a convention that is used by many file systems. It's a matter
> of
> >> > > > > > familiarity with such path names that would help users in
> using
> >> > HDFS
> >> > > > > > snapshots.
> >> > > > > >
> >> > > > > > I am including the hdfs-dev in this email. Would appreciate
> if we
> >> > > could
> >> > > > > > work together and come up with a solution.
> >> > > > > >
> >> > > > > > You can find sample output from hdfs command here:
> >> > > > > > http://pastebin.com/bBqR4Fvr
> >> > > > > >
> >> > > > > > Cheers
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > // Jonathan Hsieh (shay)
> >> > > > > // Software Engineer, Cloudera
> >> > > > > // jon@cloudera.com
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // Software Engineer, Cloudera
> >> // jon@cloudera.com
> >>
> >>
> >>
> >>
> >>
>
>
>
> --
> Harsh J
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Harsh J <ha...@cloudera.com>.
Thanks Enis and Andrew; I think I missed the key point of conformance
with other FSes' behavior.
On Wed, Apr 17, 2013 at 11:29 PM, Enis Söztutar <en...@gmail.com> wrote:
> Harsh, the discussion above includes the reasoning behind forcing the
> change in hbase rather than hdfs. Although HBase has shipped with this,
> HDFS's snapshots are user visible, meaning that you can do:
>
> hadoop fs -ls /user/foo/.snapshot/
>
> Plus, it is a convention for file systems (netapp, etc) to expose snapshots
> this way, having a name ".snapshot". HBase's snapshot directories are not
> user visible, and not widely used yet.
>
> Enis
>
>
> On Wed, Apr 17, 2013 at 9:14 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> Thanks for the consideration but we've just committed a change to address
>> this as HBASE-8352
>>
>>
>> On Wednesday, April 17, 2013, Harsh J wrote:
>>
>> > Pardon my late inquisition here but since HBase already shipped out
>> > with a name .snapshots/, why do we force them to change it, and not
>> > rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
>> > Snapshots has not been released for any users yet. The way I see it,
>> > that'd be much more easier to do than making a workaround for a done
>> > deal on HBase, which already has its snapshot's users.
>> >
>> > @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
>> > applicable to other FileSystem interface implementations as well, then
>> > .hdfs-snapshots should be fine for it - no?
>> >
>> > On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
>> > > Hi,
>> > > Please take a look at patch v5 attached to HBASE-8352.
>> > >
>> > > It would be nice to resolve this blocker today so that 0.94.7 RC can be
>> > cut.
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
>> > wrote:
>> > >
>> > >> Please see my last comment on the jira. We can make this work without
>> > >> breaking users who are using HDFS snapshots.
>> > >>
>> > >> ------------------------------
>> > >> *From:* Ted Yu <yu...@gmail.com>
>> > >> *To:* dev@hbase.apache.org
>> > >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
>> > >> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> > >> *Subject:* Re: collision in the naming of '.snapshot' directory
>> between
>> > >> hdfs snapshot and hbase snapshot
>> > >>
>> > >> Let's get proper release notes for HBASE-8352 .
>> > >>
>> > >> Either Lars or I can send out notification to user mailing list so
>> that
>> > >> there is enough preparation for this change.
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
>> > wrote:
>> > >>
>> > >> I was away from keyboard when I asserted that hdfs snapshot was a
>> hadoop
>> > >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
>> > feature.
>> > >> (I'm a little surprised -- expected this to be a hadoop2 compat
>> > breaking
>> > >> feature) -- so I agree that this is a bit more urgent.
>> > >>
>> > >> Anyway, I agree that the fs .snapshot naming convention is long
>> standing
>> > >> and should win.
>> > >>
>> > >> My concern is with breaking compatibility in 0.94 again -- if we don't
>> > go
>> > >> down the conf variable route, I consider having docs to properly
>> > document
>> > >> how to do the upgrade and caveats of doing the upgrade in the
>> > docs/release
>> > >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6
>> to
>> > >> 0.94.7, and to possibly to 0.95).
>> > >>
>> > >> Jon.
>> > >>
>> > >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >>
>> > >> > bq. Alternatively, we can detect the underlying Hadoop version, and
>> > use
>> > >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> > >> >
>> > >> > I think this would introduce more confusion, especially for
>> > operations.
>> > >> >
>> > >> > Cheers
>> > >> >
>> > >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > Because HDFS exposes the snapshots so that the normal file system
>> > >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
>> > the
>> > >> > > .snapshot name makes sense. OTOH, nothing is specific about the
>> dir
>> > >> name
>> > >> > > that is chosen by HBase.
>> > >> > >
>> > >> > > I would prefer to change the dir name in 0.94 as well, since 0.94
>> is
>> > >> also
>> > >> > > being run on top of hadoop 2. Alternatively, we can detect the
>> > >> underlying
>> > >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in
>> 0.94
>> > >> > > depending on h1 & h2.
>> > >> > >
>> > >> > > Enis
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
>> > wrote:
>> > >> > >
>> > >> > > > bq. let's make the hbase snapshot for a conf variable.
>> > >> > > >
>> > >> > > > Once we decide on the new name of snapshot directory, we should
>> > still
>> > >> > use
>> > >> > > > hardcoded value. This aligns with current code base:
>> > >> > > > See this snippet from HConstants:
>> > >--
>> > Harsh J
>> >
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
--
Harsh J
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Harsh J <ha...@cloudera.com>.
Thanks Enis and Andrew; I think I missed the key point of conformance
with other FSes' behavior.
On Wed, Apr 17, 2013 at 11:29 PM, Enis Söztutar <en...@gmail.com> wrote:
> Harsh, the discussion above includes the reasoning behind forcing the
> change in hbase rather than hdfs. Although HBase has shipped with this,
> HDFS's snapshots are user visible, meaning that you can do:
>
> hadoop fs -ls /user/foo/.snapshot/
>
> Plus, it is a convention for file systems (netapp, etc) to expose snapshots
> this way, having a name ".snapshot". HBase's snapshot directories are not
> user visible, and not widely used yet.
>
> Enis
>
>
> On Wed, Apr 17, 2013 at 9:14 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> Thanks for the consideration but we've just committed a change to address
>> this as HBASE-8352
>>
>>
>> On Wednesday, April 17, 2013, Harsh J wrote:
>>
>> > Pardon my late inquisition here but since HBase already shipped out
>> > with a name .snapshots/, why do we force them to change it, and not
>> > rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
>> > Snapshots has not been released for any users yet. The way I see it,
>> > that'd be much more easier to do than making a workaround for a done
>> > deal on HBase, which already has its snapshot's users.
>> >
>> > @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
>> > applicable to other FileSystem interface implementations as well, then
>> > .hdfs-snapshots should be fine for it - no?
>> >
>> > On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
>> > > Hi,
>> > > Please take a look at patch v5 attached to HBASE-8352.
>> > >
>> > > It would be nice to resolve this blocker today so that 0.94.7 RC can be
>> > cut.
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
>> > wrote:
>> > >
>> > >> Please see my last comment on the jira. We can make this work without
>> > >> breaking users who are using HDFS snapshots.
>> > >>
>> > >> ------------------------------
>> > >> *From:* Ted Yu <yu...@gmail.com>
>> > >> *To:* dev@hbase.apache.org
>> > >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
>> > >> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> > >> *Subject:* Re: collision in the naming of '.snapshot' directory
>> between
>> > >> hdfs snapshot and hbase snapshot
>> > >>
>> > >> Let's get proper release notes for HBASE-8352 .
>> > >>
>> > >> Either Lars or I can send out notification to user mailing list so
>> that
>> > >> there is enough preparation for this change.
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
>> > wrote:
>> > >>
>> > >> I was away from keyboard when I asserted that hdfs snapshot was a
>> hadoop
>> > >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
>> > feature.
>> > >> (I'm a little surprised -- expected this to be a hadoop2 compat
>> > breaking
>> > >> feature) -- so I agree that this is a bit more urgent.
>> > >>
>> > >> Anyway, I agree that the fs .snapshot naming convention is long
>> standing
>> > >> and should win.
>> > >>
>> > >> My concern is with breaking compatibility in 0.94 again -- if we don't
>> > go
>> > >> down the conf variable route, I consider having docs to properly
>> > document
>> > >> how to do the upgrade and caveats of doing the upgrade in the
>> > docs/release
>> > >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6
>> to
>> > >> 0.94.7, and to possibly to 0.95).
>> > >>
>> > >> Jon.
>> > >>
>> > >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >>
>> > >> > bq. Alternatively, we can detect the underlying Hadoop version, and
>> > use
>> > >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> > >> >
>> > >> > I think this would introduce more confusion, especially for
>> > operations.
>> > >> >
>> > >> > Cheers
>> > >> >
>> > >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > Because HDFS exposes the snapshots so that the normal file system
>> > >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
>> > the
>> > >> > > .snapshot name makes sense. OTOH, nothing is specific about the
>> dir
>> > >> name
>> > >> > > that is chosen by HBase.
>> > >> > >
>> > >> > > I would prefer to change the dir name in 0.94 as well, since 0.94
>> is
>> > >> also
>> > >> > > being run on top of hadoop 2. Alternatively, we can detect the
>> > >> underlying
>> > >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in
>> 0.94
>> > >> > > depending on h1 & h2.
>> > >> > >
>> > >> > > Enis
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
>> > wrote:
>> > >> > >
>> > >> > > > bq. let's make the hbase snapshot for a conf variable.
>> > >> > > >
>> > >> > > > Once we decide on the new name of snapshot directory, we should
>> > still
>> > >> > use
>> > >> > > > hardcoded value. This aligns with current code base:
>> > >> > > > See this snippet from HConstants:
>> > >--
>> > Harsh J
>> >
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
--
Harsh J
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Enis Söztutar <en...@gmail.com>.
Harsh, the discussion above includes the reasoning behind forcing the
change in hbase rather than hdfs. Although HBase has shipped with this,
HDFS's snapshots are user visible, meaning that you can do:
hadoop fs -ls /user/foo/.snapshot/
Plus, it is a convention for file systems (netapp, etc) to expose snapshots
this way, having a name ".snapshot". HBase's snapshot directories are not
user visible, and not widely used yet.
Enis
On Wed, Apr 17, 2013 at 9:14 AM, Andrew Purtell <ap...@apache.org> wrote:
> Thanks for the consideration but we've just committed a change to address
> this as HBASE-8352
>
>
> On Wednesday, April 17, 2013, Harsh J wrote:
>
> > Pardon my late inquisition here but since HBase already shipped out
> > with a name .snapshots/, why do we force them to change it, and not
> > rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> > Snapshots has not been released for any users yet. The way I see it,
> > that'd be much more easier to do than making a workaround for a done
> > deal on HBase, which already has its snapshot's users.
> >
> > @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> > applicable to other FileSystem interface implementations as well, then
> > .hdfs-snapshots should be fine for it - no?
> >
> > On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> > > Hi,
> > > Please take a look at patch v5 attached to HBASE-8352.
> > >
> > > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> > cut.
> > >
> > > Thanks
> > >
> > > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > >> Please see my last comment on the jira. We can make this work without
> > >> breaking users who are using HDFS snapshots.
> > >>
> > >> ------------------------------
> > >> *From:* Ted Yu <yu...@gmail.com>
> > >> *To:* dev@hbase.apache.org
> > >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> > >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> > >> *Subject:* Re: collision in the naming of '.snapshot' directory
> between
> > >> hdfs snapshot and hbase snapshot
> > >>
> > >> Let's get proper release notes for HBASE-8352 .
> > >>
> > >> Either Lars or I can send out notification to user mailing list so
> that
> > >> there is enough preparation for this change.
> > >>
> > >> Cheers
> > >>
> > >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
> > wrote:
> > >>
> > >> I was away from keyboard when I asserted that hdfs snapshot was a
> hadoop
> > >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
> > feature.
> > >> (I'm a little surprised -- expected this to be a hadoop2 compat
> > breaking
> > >> feature) -- so I agree that this is a bit more urgent.
> > >>
> > >> Anyway, I agree that the fs .snapshot naming convention is long
> standing
> > >> and should win.
> > >>
> > >> My concern is with breaking compatibility in 0.94 again -- if we don't
> > go
> > >> down the conf variable route, I consider having docs to properly
> > document
> > >> how to do the upgrade and caveats of doing the upgrade in the
> > docs/release
> > >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6
> to
> > >> 0.94.7, and to possibly to 0.95).
> > >>
> > >> Jon.
> > >>
> > >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> > use
> > >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> > >> >
> > >> > I think this would introduce more confusion, especially for
> > operations.
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Because HDFS exposes the snapshots so that the normal file system
> > >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> > the
> > >> > > .snapshot name makes sense. OTOH, nothing is specific about the
> dir
> > >> name
> > >> > > that is chosen by HBase.
> > >> > >
> > >> > > I would prefer to change the dir name in 0.94 as well, since 0.94
> is
> > >> also
> > >> > > being run on top of hadoop 2. Alternatively, we can detect the
> > >> underlying
> > >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in
> 0.94
> > >> > > depending on h1 & h2.
> > >> > >
> > >> > > Enis
> > >> > >
> > >> > >
> > >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > bq. let's make the hbase snapshot for a conf variable.
> > >> > > >
> > >> > > > Once we decide on the new name of snapshot directory, we should
> > still
> > >> > use
> > >> > > > hardcoded value. This aligns with current code base:
> > >> > > > See this snippet from HConstants:
> > >--
> > Harsh J
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by Tsz Wo Sze <sz...@yahoo.com>.
@HBase-dev, thanks for yielding the reserved word ".snapshot" to HDFS and the fast fix for addressing the problem. You guys have done a great job!
@Harsh, it seems that more people think that .snapshot is better.
Tsz-Wo
________________________________
From: Andrew Purtell <ap...@apache.org>
To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
Cc: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Wednesday, April 17, 2013 9:14 AM
Subject: Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Thanks for the consideration but we've just committed a change to address
this as HBASE-8352
On Wednesday, April 17, 2013, Harsh J wrote:
> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >> ------------------------------
> >> *From:* Ted Yu <yu...@gmail.com>
> >> *To:* dev@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
> feature.
> >> (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route, I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >--
> Harsh J
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for the consideration but we've just committed a change to address
this as HBASE-8352
On Wednesday, April 17, 2013, Harsh J wrote:
> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >> ------------------------------
> >> *From:* Ted Yu <yu...@gmail.com>
> >> *To:* dev@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
> feature.
> >> (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route, I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >--
> Harsh J
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for the consideration but we've just committed a change to address
this as HBASE-8352
On Wednesday, April 17, 2013, Harsh J wrote:
> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org>
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >> ------------------------------
> >> *From:* Ted Yu <yu...@gmail.com>
> >> *To:* dev@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5
> feature.
> >> (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route, I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >--
> Harsh J
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Harsh J <ha...@cloudera.com>.
Pardon my late inquisition here but since HBase already shipped out
with a name .snapshots/, why do we force them to change it, and not
rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
Snapshots has not been released for any users yet. The way I see it,
that'd be much more easier to do than making a workaround for a done
deal on HBase, which already has its snapshot's users.
@Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
applicable to other FileSystem interface implementations as well, then
.hdfs-snapshots should be fine for it - no?
On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> Please take a look at patch v5 attached to HBASE-8352.
>
> It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.
>
> Thanks
>
> On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org> wrote:
>
>> Please see my last comment on the jira. We can make this work without
>> breaking users who are using HDFS snapshots.
>>
>> ------------------------------
>> *From:* Ted Yu <yu...@gmail.com>
>> *To:* dev@hbase.apache.org
>> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
>> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> *Subject:* Re: collision in the naming of '.snapshot' directory between
>> hdfs snapshot and hbase snapshot
>>
>> Let's get proper release notes for HBASE-8352 .
>>
>> Either Lars or I can send out notification to user mailing list so that
>> there is enough preparation for this change.
>>
>> Cheers
>>
>> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>
>> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
>> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>> feature) -- so I agree that this is a bit more urgent.
>>
>> Anyway, I agree that the fs .snapshot naming convention is long standing
>> and should win.
>>
>> My concern is with breaking compatibility in 0.94 again -- if we don't go
>> down the conf variable route, I consider having docs to properly document
>> how to do the upgrade and caveats of doing the upgrade in the docs/release
>> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
>> 0.94.7, and to possibly to 0.95).
>>
>> Jon.
>>
>> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > bq. Alternatively, we can detect the underlying Hadoop version, and use
>> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> >
>> > I think this would introduce more confusion, especially for operations.
>> >
>> > Cheers
>> >
>> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
>> wrote:
>> >
>> > > Because HDFS exposes the snapshots so that the normal file system
>> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
>> name
>> > > that is chosen by HBase.
>> > >
>> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
>> also
>> > > being run on top of hadoop 2. Alternatively, we can detect the
>> underlying
>> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > > depending on h1 & h2.
>> > >
>> > > Enis
>> > >
>> > >
>> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > > > bq. let's make the hbase snapshot for a conf variable.
>> > > >
>> > > > Once we decide on the new name of snapshot directory, we should still
>> > use
>> > > > hardcoded value. This aligns with current code base:
>> > > > See this snippet from HConstants:
>> > > >
>> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
>> > > >
>> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > > HREGION_LOGDIR_NAME,
>> > > >
>> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > > >
>> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> > SNAPSHOT_DIR_NAME,
>> > > > HBASE_TEMP_DIRECTORY }));
>> > > > Cheers
>> > > >
>> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> > > wrote:
>> > > >
>> > > > > constraints:
>> > > > >
>> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> > will
>> > > > ever
>> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> > > current
>> > > > A
>> > > > > pache Hbase 0.94.x's.
>> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > > should
>> > > > > pick a different table snapshot name to respect fs conventions.
>> > > > >
>> > > > > proposed actions:
>> > > > >
>> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > > 2) let's publish instructions on how to update the hbase snapshot
>> > dir:
>> > > > > shutdown hbase, config update, rename dir, restart hbase.
>> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
>> > 94
>> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
>> > require
>> > > > > shutdown and update scripts so it seems like the ideal time to
>> > > autoforce
>> > > > > this default change.
>> > > > >
>> > > > > Thoughts?
>> > > > >
>> > > > >
>> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > > > >
>> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
>> > > 0.94.7.
>> > > > > >
>> > > > > > -- Lars
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > ________________________________
>> > > > > > From: Ted Yu <yu...@gmail.com>
>> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > > > Subject: collision in the naming of '.snapshot' directory between
>> > > hdfs
>> > > > > > snapshot and hbase snapshot
>> > > > > >
>> > > > > >
>> > > > > > Hi,
>> > > > > > This afternoon Huned ad I discovered an issue while playing with
>> > > HBase
>> > > > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > > > >
>> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> > .snapshot
>> > > > as
>> > > > > a
>> > > > > > component while HBase tries to create paths with .snapshot as a
>> > > > > component.
>> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> > give
>> > > > up
>> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
>> in
>> > > > > 0.94.6
>> > > > > > (quite recently) and it may not be too late to change HBase to
>> use
>> > a
>> > > > > > different path component in an upcoming new release.
>> > > > > >
>> > > > > > In HBase these path names are not user visible. If there is a
>> > > > deployment
>> > > > > of
>> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
>> > to
>> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
>> > > release
>> > > > > of
>> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
>> name
>> > > and
>> > > > > is
>> > > > > > a convention that is used by many file systems. It's a matter of
>> > > > > > familiarity with such path names that would help users in using
>> > HDFS
>> > > > > > snapshots.
>> > > > > >
>> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
>> > > could
>> > > > > > work together and come up with a solution.
>> > > > > >
>> > > > > > You can find sample output from hdfs command here:
>> > > > > > http://pastebin.com/bBqR4Fvr
>> > > > > >
>> > > > > > Cheers
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > // Jonathan Hsieh (shay)
>> > > > > // Software Engineer, Cloudera
>> > > > > // jon@cloudera.com
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
>>
>>
>>
>>
--
Harsh J
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Harsh J <ha...@cloudera.com>.
Pardon my late inquisition here but since HBase already shipped out
with a name .snapshots/, why do we force them to change it, and not
rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
Snapshots has not been released for any users yet. The way I see it,
that'd be much more easier to do than making a workaround for a done
deal on HBase, which already has its snapshot's users.
@Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
applicable to other FileSystem interface implementations as well, then
.hdfs-snapshots should be fine for it - no?
On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> Please take a look at patch v5 attached to HBASE-8352.
>
> It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.
>
> Thanks
>
> On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org> wrote:
>
>> Please see my last comment on the jira. We can make this work without
>> breaking users who are using HDFS snapshots.
>>
>> ------------------------------
>> *From:* Ted Yu <yu...@gmail.com>
>> *To:* dev@hbase.apache.org
>> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
>> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> *Subject:* Re: collision in the naming of '.snapshot' directory between
>> hdfs snapshot and hbase snapshot
>>
>> Let's get proper release notes for HBASE-8352 .
>>
>> Either Lars or I can send out notification to user mailing list so that
>> there is enough preparation for this change.
>>
>> Cheers
>>
>> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>
>> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
>> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>> feature) -- so I agree that this is a bit more urgent.
>>
>> Anyway, I agree that the fs .snapshot naming convention is long standing
>> and should win.
>>
>> My concern is with breaking compatibility in 0.94 again -- if we don't go
>> down the conf variable route, I consider having docs to properly document
>> how to do the upgrade and caveats of doing the upgrade in the docs/release
>> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
>> 0.94.7, and to possibly to 0.95).
>>
>> Jon.
>>
>> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > bq. Alternatively, we can detect the underlying Hadoop version, and use
>> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> >
>> > I think this would introduce more confusion, especially for operations.
>> >
>> > Cheers
>> >
>> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
>> wrote:
>> >
>> > > Because HDFS exposes the snapshots so that the normal file system
>> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
>> name
>> > > that is chosen by HBase.
>> > >
>> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
>> also
>> > > being run on top of hadoop 2. Alternatively, we can detect the
>> underlying
>> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > > depending on h1 & h2.
>> > >
>> > > Enis
>> > >
>> > >
>> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > > > bq. let's make the hbase snapshot for a conf variable.
>> > > >
>> > > > Once we decide on the new name of snapshot directory, we should still
>> > use
>> > > > hardcoded value. This aligns with current code base:
>> > > > See this snippet from HConstants:
>> > > >
>> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
>> > > >
>> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > > HREGION_LOGDIR_NAME,
>> > > >
>> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > > >
>> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> > SNAPSHOT_DIR_NAME,
>> > > > HBASE_TEMP_DIRECTORY }));
>> > > > Cheers
>> > > >
>> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> > > wrote:
>> > > >
>> > > > > constraints:
>> > > > >
>> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> > will
>> > > > ever
>> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> > > current
>> > > > A
>> > > > > pache Hbase 0.94.x's.
>> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > > should
>> > > > > pick a different table snapshot name to respect fs conventions.
>> > > > >
>> > > > > proposed actions:
>> > > > >
>> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > > 2) let's publish instructions on how to update the hbase snapshot
>> > dir:
>> > > > > shutdown hbase, config update, rename dir, restart hbase.
>> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
>> > 94
>> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
>> > require
>> > > > > shutdown and update scripts so it seems like the ideal time to
>> > > autoforce
>> > > > > this default change.
>> > > > >
>> > > > > Thoughts?
>> > > > >
>> > > > >
>> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > > > >
>> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
>> > > 0.94.7.
>> > > > > >
>> > > > > > -- Lars
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > ________________________________
>> > > > > > From: Ted Yu <yu...@gmail.com>
>> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > > > Subject: collision in the naming of '.snapshot' directory between
>> > > hdfs
>> > > > > > snapshot and hbase snapshot
>> > > > > >
>> > > > > >
>> > > > > > Hi,
>> > > > > > This afternoon Huned ad I discovered an issue while playing with
>> > > HBase
>> > > > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > > > >
>> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> > .snapshot
>> > > > as
>> > > > > a
>> > > > > > component while HBase tries to create paths with .snapshot as a
>> > > > > component.
>> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> > give
>> > > > up
>> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
>> in
>> > > > > 0.94.6
>> > > > > > (quite recently) and it may not be too late to change HBase to
>> use
>> > a
>> > > > > > different path component in an upcoming new release.
>> > > > > >
>> > > > > > In HBase these path names are not user visible. If there is a
>> > > > deployment
>> > > > > of
>> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
>> > to
>> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
>> > > release
>> > > > > of
>> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
>> name
>> > > and
>> > > > > is
>> > > > > > a convention that is used by many file systems. It's a matter of
>> > > > > > familiarity with such path names that would help users in using
>> > HDFS
>> > > > > > snapshots.
>> > > > > >
>> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
>> > > could
>> > > > > > work together and come up with a solution.
>> > > > > >
>> > > > > > You can find sample output from hdfs command here:
>> > > > > > http://pastebin.com/bBqR4Fvr
>> > > > > >
>> > > > > > Cheers
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > // Jonathan Hsieh (shay)
>> > > > > // Software Engineer, Cloudera
>> > > > > // jon@cloudera.com
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
>>
>>
>>
>>
--
Harsh J
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Hi,
Please take a look at patch v5 attached to HBASE-8352.
It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.
Thanks
On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org> wrote:
> Please see my last comment on the jira. We can make this work without
> breaking users who are using HDFS snapshots.
>
> ------------------------------
> *From:* Ted Yu <yu...@gmail.com>
> *To:* dev@hbase.apache.org
> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> *Sent:* Tuesday, April 16, 2013 10:00 AM
> *Subject:* Re: collision in the naming of '.snapshot' directory between
> hdfs snapshot and hbase snapshot
>
> Let's get proper release notes for HBASE-8352 .
>
> Either Lars or I can send out notification to user mailing list so that
> there is enough preparation for this change.
>
> Cheers
>
> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
> feature) -- so I agree that this is a bit more urgent.
>
> Anyway, I agree that the fs .snapshot naming convention is long standing
> and should win.
>
> My concern is with breaking compatibility in 0.94 again -- if we don't go
> down the conf variable route, I consider having docs to properly document
> how to do the upgrade and caveats of doing the upgrade in the docs/release
> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> 0.94.7, and to possibly to 0.95).
>
> Jon.
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Alternatively, we can detect the underlying Hadoop version, and use
> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >
> > I think this would introduce more confusion, especially for operations.
> >
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >
> > > Because HDFS exposes the snapshots so that the normal file system
> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> name
> > > that is chosen by HBase.
> > >
> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> also
> > > being run on top of hadoop 2. Alternatively, we can detect the
> underlying
> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> > > depending on h1 & h2.
> > >
> > > Enis
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. let's make the hbase snapshot for a conf variable.
> > > >
> > > > Once we decide on the new name of snapshot directory, we should still
> > use
> > > > hardcoded value. This aligns with current code base:
> > > > See this snippet from HConstants:
> > > >
> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
> > > >
> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> > > > HREGION_LOGDIR_NAME,
> > > >
> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> > > >
> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> > SNAPSHOT_DIR_NAME,
> > > > HBASE_TEMP_DIRECTORY }));
> > > > Cheers
> > > >
> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > > > constraints:
> > > > >
> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
> > will
> > > > ever
> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> > > current
> > > > A
> > > > > pache Hbase 0.94.x's.
> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> > > should
> > > > > pick a different table snapshot name to respect fs conventions.
> > > > >
> > > > > proposed actions:
> > > > >
> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > > > 2) let's publish instructions on how to update the hbase snapshot
> > dir:
> > > > > shutdown hbase, config update, rename dir, restart hbase.
> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
> > 94
> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> > require
> > > > > shutdown and update scripts so it seems like the ideal time to
> > > autoforce
> > > > > this default change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
> > > > >
> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
> > > 0.94.7.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > > > Subject: collision in the naming of '.snapshot' directory between
> > > hdfs
> > > > > > snapshot and hbase snapshot
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > This afternoon Huned ad I discovered an issue while playing with
> > > HBase
> > > > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > > > >
> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> > .snapshot
> > > > as
> > > > > a
> > > > > > component while HBase tries to create paths with .snapshot as a
> > > > > component.
> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
> > give
> > > > up
> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
> in
> > > > > 0.94.6
> > > > > > (quite recently) and it may not be too late to change HBase to
> use
> > a
> > > > > > different path component in an upcoming new release.
> > > > > >
> > > > > > In HBase these path names are not user visible. If there is a
> > > > deployment
> > > > > of
> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
> > to
> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
> > > release
> > > > > of
> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
> name
> > > and
> > > > > is
> > > > > > a convention that is used by many file systems. It's a matter of
> > > > > > familiarity with such path names that would help users in using
> > HDFS
> > > > > > snapshots.
> > > > > >
> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
> > > could
> > > > > > work together and come up with a solution.
> > > > > >
> > > > > > You can find sample output from hdfs command here:
> > > > > > http://pastebin.com/bBqR4Fvr
> > > > > >
> > > > > > Cheers
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // Software Engineer, Cloudera
> > > > > // jon@cloudera.com
> > > > >
> > > >
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Hi,
Please take a look at patch v5 attached to HBASE-8352.
It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.
Thanks
On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <la...@apache.org> wrote:
> Please see my last comment on the jira. We can make this work without
> breaking users who are using HDFS snapshots.
>
> ------------------------------
> *From:* Ted Yu <yu...@gmail.com>
> *To:* dev@hbase.apache.org
> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
> *Sent:* Tuesday, April 16, 2013 10:00 AM
> *Subject:* Re: collision in the naming of '.snapshot' directory between
> hdfs snapshot and hbase snapshot
>
> Let's get proper release notes for HBASE-8352 .
>
> Either Lars or I can send out notification to user mailing list so that
> there is enough preparation for this change.
>
> Cheers
>
> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
> feature) -- so I agree that this is a bit more urgent.
>
> Anyway, I agree that the fs .snapshot naming convention is long standing
> and should win.
>
> My concern is with breaking compatibility in 0.94 again -- if we don't go
> down the conf variable route, I consider having docs to properly document
> how to do the upgrade and caveats of doing the upgrade in the docs/release
> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> 0.94.7, and to possibly to 0.95).
>
> Jon.
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Alternatively, we can detect the underlying Hadoop version, and use
> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >
> > I think this would introduce more confusion, especially for operations.
> >
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >
> > > Because HDFS exposes the snapshots so that the normal file system
> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> name
> > > that is chosen by HBase.
> > >
> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> also
> > > being run on top of hadoop 2. Alternatively, we can detect the
> underlying
> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> > > depending on h1 & h2.
> > >
> > > Enis
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. let's make the hbase snapshot for a conf variable.
> > > >
> > > > Once we decide on the new name of snapshot directory, we should still
> > use
> > > > hardcoded value. This aligns with current code base:
> > > > See this snippet from HConstants:
> > > >
> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
> > > >
> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> > > > HREGION_LOGDIR_NAME,
> > > >
> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> > > >
> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> > SNAPSHOT_DIR_NAME,
> > > > HBASE_TEMP_DIRECTORY }));
> > > > Cheers
> > > >
> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > > > constraints:
> > > > >
> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
> > will
> > > > ever
> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> > > current
> > > > A
> > > > > pache Hbase 0.94.x's.
> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> > > should
> > > > > pick a different table snapshot name to respect fs conventions.
> > > > >
> > > > > proposed actions:
> > > > >
> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > > > 2) let's publish instructions on how to update the hbase snapshot
> > dir:
> > > > > shutdown hbase, config update, rename dir, restart hbase.
> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
> > 94
> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> > require
> > > > > shutdown and update scripts so it seems like the ideal time to
> > > autoforce
> > > > > this default change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
> > > > >
> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
> > > 0.94.7.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > > > Subject: collision in the naming of '.snapshot' directory between
> > > hdfs
> > > > > > snapshot and hbase snapshot
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > This afternoon Huned ad I discovered an issue while playing with
> > > HBase
> > > > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > > > >
> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> > .snapshot
> > > > as
> > > > > a
> > > > > > component while HBase tries to create paths with .snapshot as a
> > > > > component.
> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
> > give
> > > > up
> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
> in
> > > > > 0.94.6
> > > > > > (quite recently) and it may not be too late to change HBase to
> use
> > a
> > > > > > different path component in an upcoming new release.
> > > > > >
> > > > > > In HBase these path names are not user visible. If there is a
> > > > deployment
> > > > > of
> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
> > to
> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
> > > release
> > > > > of
> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
> name
> > > and
> > > > > is
> > > > > > a convention that is used by many file systems. It's a matter of
> > > > > > familiarity with such path names that would help users in using
> > HDFS
> > > > > > snapshots.
> > > > > >
> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
> > > could
> > > > > > work together and come up with a solution.
> > > > > >
> > > > > > You can find sample output from hdfs command here:
> > > > > > http://pastebin.com/bBqR4Fvr
> > > > > >
> > > > > > Cheers
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // Software Engineer, Cloudera
> > > > > // jon@cloudera.com
> > > > >
> > > >
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
Please see my last comment on the jira. We can make this work without breaking users who are using HDFS snapshots.
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org
Cc: hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
Sent: Tuesday, April 16, 2013 10:00 AM
Subject: Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Let's get proper release notes for HBASE-8352 .
Either Lars or I can send out notification to user mailing list so that there is enough preparation for this change.
Cheers
On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>feature) -- so I agree that this is a bit more urgent.
>
>Anyway, I agree that the fs .snapshot naming convention is long standing
>and should win.
>
>My concern is with breaking compatibility in 0.94 again -- if we don't go
>down the conf variable route, I consider having docs to properly document
>how to do the upgrade and caveats of doing the upgrade in the docs/release
>notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
>0.94.7, and to possibly to 0.95).
>
>Jon.
>
>On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. Alternatively, we can detect the underlying Hadoop version, and use
>
>> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>>
>> I think this would introduce more confusion, especially for operations.
>>
>> Cheers
>>
>> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
>>
>> > Because HDFS exposes the snapshots so that the normal file system
>> > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > .snapshot name makes sense. OTOH, nothing is specific about the dir name
>> > that is chosen by HBase.
>> >
>> > I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>> > being run on top of hadoop 2. Alternatively, we can detect the underlying
>> > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > depending on h1 & h2.
>> >
>> > Enis
>> >
>> >
>> > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > bq. let's make the hbase snapshot for a conf variable.
>> > >
>> > > Once we decide on the new name of snapshot directory, we should still
>> use
>> > > hardcoded value. This aligns with current code base:
>> > > See this snippet from HConstants:
>> > >
>> > > public static final List<String> HBASE_NON_TABLE_DIRS =
>> > >
>> > > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > HREGION_LOGDIR_NAME,
>> > >
>> > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > >
>> > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> SNAPSHOT_DIR_NAME,
>> > > HBASE_TEMP_DIRECTORY }));
>> > > Cheers
>> > >
>> > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> > wrote:
>> > >
>> > > > constraints:
>> > > >
>> > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> will
>> > > ever
>> > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> > current
>> > > A
>> > > > pache Hbase 0.94.x's.
>> > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > should
>> > > > pick a different table snapshot name to respect fs conventions.
>> > > >
>> > > > proposed actions:
>> > > >
>> > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > 2) let's publish instructions on how to update the hbase snapshot
>> dir:
>> > > > shutdown hbase, config update, rename dir, restart hbase.
>> > > > 3) I lean towards leaving the current default hbase snapshot dir in
>> 94
>> > > > since it shouldn't be affected. upgrading hbase to 95/96 will
>> require
>> > > > shutdown and update scripts so it seems like the ideal time to
>> > autoforce
>> > > > this default change.
>> > > >
>> > > > Thoughts?
>> > > >
>> > > >
>> > > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > > >
>> > > > > OK. Let's try to fix that quickly, so that I can release HBase
>> > 0.94.7.
>> > > > >
>> > > > > -- Lars
>> > > > >
>> > > > >
>> > > > >
>> > > > > ________________________________
>> > > > > From: Ted Yu <yu...@gmail.com>
>> > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > > Subject: collision in the naming of '.snapshot' directory between
>> > hdfs
>> > > > > snapshot and hbase snapshot
>> > > > >
>> > > > >
>> > > > > Hi,
>> > > > > This afternoon Huned ad I discovered an issue while playing with
>> > HBase
>> > > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > > >
>> > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> .snapshot
>> > > as
>> > > > a
>> > > > > component while HBase tries to create paths with .snapshot as a
>> > > > component.
>> > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> give
>> > > up
>> > > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>> > > > 0.94.6
>> > > > > (quite recently) and it may not be too late to change HBase to use
>> a
>> > > > > different path component in an upcoming new release.
>> > > > >
>> > > > > In HBase these path names are not user visible. If there is a
>> > > deployment
>> > > > of
>> > > > > 0.94.6, one could provide a migration tool that renames .snapshot
>> to
>> > > > > .hbase-snapshot or something to be able to move to the Snapshot
>> > release
>> > > > of
>> > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
>> > and
>> > > > is
>> > > > > a convention that is used by many file systems. It's a matter of
>> > > > > familiarity with such path names that would help users in using
>> HDFS
>> > > > > snapshots.
>> > > > >
>> > > > > I am including the hdfs-dev in this email. Would appreciate if we
>> > could
>> > > > > work together and come up with a solution.
>> > > > >
>> > > > > You can find sample output from hdfs command here:
>> > > > > http://pastebin.com/bBqR4Fvr
>> > > > >
>> > > > > Cheers
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > // Jonathan Hsieh (shay)
>> > > > // Software Engineer, Cloudera
>> > > > // jon@cloudera.com
>> > > >
>> > >
>> >
>>
>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
Please see my last comment on the jira. We can make this work without breaking users who are using HDFS snapshots.
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org
Cc: hdfs-dev@hadoop.apache.org; lars hofhansl <la...@apache.org>
Sent: Tuesday, April 16, 2013 10:00 AM
Subject: Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Let's get proper release notes for HBASE-8352 .
Either Lars or I can send out notification to user mailing list so that there is enough preparation for this change.
Cheers
On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>feature) -- so I agree that this is a bit more urgent.
>
>Anyway, I agree that the fs .snapshot naming convention is long standing
>and should win.
>
>My concern is with breaking compatibility in 0.94 again -- if we don't go
>down the conf variable route, I consider having docs to properly document
>how to do the upgrade and caveats of doing the upgrade in the docs/release
>notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
>0.94.7, and to possibly to 0.95).
>
>Jon.
>
>On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. Alternatively, we can detect the underlying Hadoop version, and use
>
>> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>>
>> I think this would introduce more confusion, especially for operations.
>>
>> Cheers
>>
>> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
>>
>> > Because HDFS exposes the snapshots so that the normal file system
>> > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > .snapshot name makes sense. OTOH, nothing is specific about the dir name
>> > that is chosen by HBase.
>> >
>> > I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>> > being run on top of hadoop 2. Alternatively, we can detect the underlying
>> > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > depending on h1 & h2.
>> >
>> > Enis
>> >
>> >
>> > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > bq. let's make the hbase snapshot for a conf variable.
>> > >
>> > > Once we decide on the new name of snapshot directory, we should still
>> use
>> > > hardcoded value. This aligns with current code base:
>> > > See this snippet from HConstants:
>> > >
>> > > public static final List<String> HBASE_NON_TABLE_DIRS =
>> > >
>> > > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > HREGION_LOGDIR_NAME,
>> > >
>> > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > >
>> > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> SNAPSHOT_DIR_NAME,
>> > > HBASE_TEMP_DIRECTORY }));
>> > > Cheers
>> > >
>> > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> > wrote:
>> > >
>> > > > constraints:
>> > > >
>> > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> will
>> > > ever
>> > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> > current
>> > > A
>> > > > pache Hbase 0.94.x's.
>> > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > should
>> > > > pick a different table snapshot name to respect fs conventions.
>> > > >
>> > > > proposed actions:
>> > > >
>> > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > 2) let's publish instructions on how to update the hbase snapshot
>> dir:
>> > > > shutdown hbase, config update, rename dir, restart hbase.
>> > > > 3) I lean towards leaving the current default hbase snapshot dir in
>> 94
>> > > > since it shouldn't be affected. upgrading hbase to 95/96 will
>> require
>> > > > shutdown and update scripts so it seems like the ideal time to
>> > autoforce
>> > > > this default change.
>> > > >
>> > > > Thoughts?
>> > > >
>> > > >
>> > > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > > >
>> > > > > OK. Let's try to fix that quickly, so that I can release HBase
>> > 0.94.7.
>> > > > >
>> > > > > -- Lars
>> > > > >
>> > > > >
>> > > > >
>> > > > > ________________________________
>> > > > > From: Ted Yu <yu...@gmail.com>
>> > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > > Subject: collision in the naming of '.snapshot' directory between
>> > hdfs
>> > > > > snapshot and hbase snapshot
>> > > > >
>> > > > >
>> > > > > Hi,
>> > > > > This afternoon Huned ad I discovered an issue while playing with
>> > HBase
>> > > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > > >
>> > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> .snapshot
>> > > as
>> > > > a
>> > > > > component while HBase tries to create paths with .snapshot as a
>> > > > component.
>> > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> give
>> > > up
>> > > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>> > > > 0.94.6
>> > > > > (quite recently) and it may not be too late to change HBase to use
>> a
>> > > > > different path component in an upcoming new release.
>> > > > >
>> > > > > In HBase these path names are not user visible. If there is a
>> > > deployment
>> > > > of
>> > > > > 0.94.6, one could provide a migration tool that renames .snapshot
>> to
>> > > > > .hbase-snapshot or something to be able to move to the Snapshot
>> > release
>> > > > of
>> > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
>> > and
>> > > > is
>> > > > > a convention that is used by many file systems. It's a matter of
>> > > > > familiarity with such path names that would help users in using
>> HDFS
>> > > > > snapshots.
>> > > > >
>> > > > > I am including the hdfs-dev in this email. Would appreciate if we
>> > could
>> > > > > work together and come up with a solution.
>> > > > >
>> > > > > You can find sample output from hdfs command here:
>> > > > > http://pastebin.com/bBqR4Fvr
>> > > > >
>> > > > > Cheers
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > // Jonathan Hsieh (shay)
>> > > > // Software Engineer, Cloudera
>> > > > // jon@cloudera.com
>> > > >
>> > >
>> >
>>
>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Let's get proper release notes for HBASE-8352 .
Either Lars or I can send out notification to user mailing list so that
there is enough preparation for this change.
Cheers
On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
> feature) -- so I agree that this is a bit more urgent.
>
> Anyway, I agree that the fs .snapshot naming convention is long standing
> and should win.
>
> My concern is with breaking compatibility in 0.94 again -- if we don't go
> down the conf variable route, I consider having docs to properly document
> how to do the upgrade and caveats of doing the upgrade in the docs/release
> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> 0.94.7, and to possibly to 0.95).
>
> Jon.
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Alternatively, we can detect the underlying Hadoop version, and use
> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >
> > I think this would introduce more confusion, especially for operations.
> >
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >
> > > Because HDFS exposes the snapshots so that the normal file system
> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> name
> > > that is chosen by HBase.
> > >
> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> also
> > > being run on top of hadoop 2. Alternatively, we can detect the
> underlying
> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> > > depending on h1 & h2.
> > >
> > > Enis
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. let's make the hbase snapshot for a conf variable.
> > > >
> > > > Once we decide on the new name of snapshot directory, we should still
> > use
> > > > hardcoded value. This aligns with current code base:
> > > > See this snippet from HConstants:
> > > >
> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
> > > >
> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> > > > HREGION_LOGDIR_NAME,
> > > >
> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> > > >
> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> > SNAPSHOT_DIR_NAME,
> > > > HBASE_TEMP_DIRECTORY }));
> > > > Cheers
> > > >
> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > > > constraints:
> > > > >
> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
> > will
> > > > ever
> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> > > current
> > > > A
> > > > > pache Hbase 0.94.x's.
> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> > > should
> > > > > pick a different table snapshot name to respect fs conventions.
> > > > >
> > > > > proposed actions:
> > > > >
> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > > > 2) let's publish instructions on how to update the hbase snapshot
> > dir:
> > > > > shutdown hbase, config update, rename dir, restart hbase.
> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
> > 94
> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> > require
> > > > > shutdown and update scripts so it seems like the ideal time to
> > > autoforce
> > > > > this default change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
> > > > >
> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
> > > 0.94.7.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > > > Subject: collision in the naming of '.snapshot' directory between
> > > hdfs
> > > > > > snapshot and hbase snapshot
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > This afternoon Huned ad I discovered an issue while playing with
> > > HBase
> > > > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > > > >
> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> > .snapshot
> > > > as
> > > > > a
> > > > > > component while HBase tries to create paths with .snapshot as a
> > > > > component.
> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
> > give
> > > > up
> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
> in
> > > > > 0.94.6
> > > > > > (quite recently) and it may not be too late to change HBase to
> use
> > a
> > > > > > different path component in an upcoming new release.
> > > > > >
> > > > > > In HBase these path names are not user visible. If there is a
> > > > deployment
> > > > > of
> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
> > to
> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
> > > release
> > > > > of
> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
> name
> > > and
> > > > > is
> > > > > > a convention that is used by many file systems. It's a matter of
> > > > > > familiarity with such path names that would help users in using
> > HDFS
> > > > > > snapshots.
> > > > > >
> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
> > > could
> > > > > > work together and come up with a solution.
> > > > > >
> > > > > > You can find sample output from hdfs command here:
> > > > > > http://pastebin.com/bBqR4Fvr
> > > > > >
> > > > > > Cheers
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // Software Engineer, Cloudera
> > > > > // jon@cloudera.com
> > > > >
> > > >
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Let's get proper release notes for HBASE-8352 .
Either Lars or I can send out notification to user mailing list so that
there is enough preparation for this change.
Cheers
On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
> (I'm a little surprised -- expected this to be a hadoop2 compat breaking
> feature) -- so I agree that this is a bit more urgent.
>
> Anyway, I agree that the fs .snapshot naming convention is long standing
> and should win.
>
> My concern is with breaking compatibility in 0.94 again -- if we don't go
> down the conf variable route, I consider having docs to properly document
> how to do the upgrade and caveats of doing the upgrade in the docs/release
> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
> 0.94.7, and to possibly to 0.95).
>
> Jon.
>
> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Alternatively, we can detect the underlying Hadoop version, and use
> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >
> > I think this would introduce more confusion, especially for operations.
> >
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >
> > > Because HDFS exposes the snapshots so that the normal file system
> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> name
> > > that is chosen by HBase.
> > >
> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> also
> > > being run on top of hadoop 2. Alternatively, we can detect the
> underlying
> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> > > depending on h1 & h2.
> > >
> > > Enis
> > >
> > >
> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. let's make the hbase snapshot for a conf variable.
> > > >
> > > > Once we decide on the new name of snapshot directory, we should still
> > use
> > > > hardcoded value. This aligns with current code base:
> > > > See this snippet from HConstants:
> > > >
> > > > public static final List<String> HBASE_NON_TABLE_DIRS =
> > > >
> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> > > > HREGION_LOGDIR_NAME,
> > > >
> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> > > >
> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> > SNAPSHOT_DIR_NAME,
> > > > HBASE_TEMP_DIRECTORY }));
> > > > Cheers
> > > >
> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > > > constraints:
> > > > >
> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
> > will
> > > > ever
> > > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> > > current
> > > > A
> > > > > pache Hbase 0.94.x's.
> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> > > should
> > > > > pick a different table snapshot name to respect fs conventions.
> > > > >
> > > > > proposed actions:
> > > > >
> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > > > 2) let's publish instructions on how to update the hbase snapshot
> > dir:
> > > > > shutdown hbase, config update, rename dir, restart hbase.
> > > > > 3) I lean towards leaving the current default hbase snapshot dir in
> > 94
> > > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> > require
> > > > > shutdown and update scripts so it seems like the ideal time to
> > > autoforce
> > > > > this default change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Monday, April 15, 2013, lars hofhansl wrote:
> > > > >
> > > > > > OK. Let's try to fix that quickly, so that I can release HBase
> > > 0.94.7.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > > From: Ted Yu <yu...@gmail.com>
> > > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > > > Subject: collision in the naming of '.snapshot' directory between
> > > hdfs
> > > > > > snapshot and hbase snapshot
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > This afternoon Huned ad I discovered an issue while playing with
> > > HBase
> > > > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > > > >
> > > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> > .snapshot
> > > > as
> > > > > a
> > > > > > component while HBase tries to create paths with .snapshot as a
> > > > > component.
> > > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
> > give
> > > > up
> > > > > > the .snapshot reserved keyword. HBase released Snapshots feature
> in
> > > > > 0.94.6
> > > > > > (quite recently) and it may not be too late to change HBase to
> use
> > a
> > > > > > different path component in an upcoming new release.
> > > > > >
> > > > > > In HBase these path names are not user visible. If there is a
> > > > deployment
> > > > > of
> > > > > > 0.94.6, one could provide a migration tool that renames .snapshot
> > to
> > > > > > .hbase-snapshot or something to be able to move to the Snapshot
> > > release
> > > > > of
> > > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible
> name
> > > and
> > > > > is
> > > > > > a convention that is used by many file systems. It's a matter of
> > > > > > familiarity with such path names that would help users in using
> > HDFS
> > > > > > snapshots.
> > > > > >
> > > > > > I am including the hdfs-dev in this email. Would appreciate if we
> > > could
> > > > > > work together and come up with a solution.
> > > > > >
> > > > > > You can find sample output from hdfs command here:
> > > > > > http://pastebin.com/bBqR4Fvr
> > > > > >
> > > > > > Cheers
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // Software Engineer, Cloudera
> > > > > // jon@cloudera.com
> > > > >
> > > >
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I was away from keyboard when I asserted that hdfs snapshot was a hadoop
2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.
(I'm a little surprised -- expected this to be a hadoop2 compat breaking
feature) -- so I agree that this is a bit more urgent.
Anyway, I agree that the fs .snapshot naming convention is long standing
and should win.
My concern is with breaking compatibility in 0.94 again -- if we don't go
down the conf variable route, I consider having docs to properly document
how to do the upgrade and caveats of doing the upgrade in the docs/release
notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to
0.94.7, and to possibly to 0.95).
Jon.
On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Alternatively, we can detect the underlying Hadoop version, and use
> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>
> I think this would introduce more confusion, especially for operations.
>
> Cheers
>
> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
>
> > Because HDFS exposes the snapshots so that the normal file system
> > operations are mapped inside snapshot dirs, I think HDFS reserving the
> > .snapshot name makes sense. OTOH, nothing is specific about the dir name
> > that is chosen by HBase.
> >
> > I would prefer to change the dir name in 0.94 as well, since 0.94 is also
> > being run on top of hadoop 2. Alternatively, we can detect the underlying
> > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> > depending on h1 & h2.
> >
> > Enis
> >
> >
> > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. let's make the hbase snapshot for a conf variable.
> > >
> > > Once we decide on the new name of snapshot directory, we should still
> use
> > > hardcoded value. This aligns with current code base:
> > > See this snippet from HConstants:
> > >
> > > public static final List<String> HBASE_NON_TABLE_DIRS =
> > >
> > > Collections.unmodifiableList(Arrays.asList(new String[] {
> > > HREGION_LOGDIR_NAME,
> > >
> > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> > >
> > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> SNAPSHOT_DIR_NAME,
> > > HBASE_TEMP_DIRECTORY }));
> > > Cheers
> > >
> > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> > wrote:
> > >
> > > > constraints:
> > > >
> > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
> will
> > > ever
> > > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> > current
> > > A
> > > > pache Hbase 0.94.x's.
> > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> > should
> > > > pick a different table snapshot name to respect fs conventions.
> > > >
> > > > proposed actions:
> > > >
> > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > > 2) let's publish instructions on how to update the hbase snapshot
> dir:
> > > > shutdown hbase, config update, rename dir, restart hbase.
> > > > 3) I lean towards leaving the current default hbase snapshot dir in
> 94
> > > > since it shouldn't be affected. upgrading hbase to 95/96 will
> require
> > > > shutdown and update scripts so it seems like the ideal time to
> > autoforce
> > > > this default change.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Monday, April 15, 2013, lars hofhansl wrote:
> > > >
> > > > > OK. Let's try to fix that quickly, so that I can release HBase
> > 0.94.7.
> > > > >
> > > > > -- Lars
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: Ted Yu <yu...@gmail.com>
> > > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > > Subject: collision in the naming of '.snapshot' directory between
> > hdfs
> > > > > snapshot and hbase snapshot
> > > > >
> > > > >
> > > > > Hi,
> > > > > This afternoon Huned ad I discovered an issue while playing with
> > HBase
> > > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > > >
> > > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
> .snapshot
> > > as
> > > > a
> > > > > component while HBase tries to create paths with .snapshot as a
> > > > component.
> > > > > This leads to issues in HBase, and one of HDFS or HBase needs to
> give
> > > up
> > > > > the .snapshot reserved keyword. HBase released Snapshots feature in
> > > > 0.94.6
> > > > > (quite recently) and it may not be too late to change HBase to use
> a
> > > > > different path component in an upcoming new release.
> > > > >
> > > > > In HBase these path names are not user visible. If there is a
> > > deployment
> > > > of
> > > > > 0.94.6, one could provide a migration tool that renames .snapshot
> to
> > > > > .hbase-snapshot or something to be able to move to the Snapshot
> > release
> > > > of
> > > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
> > and
> > > > is
> > > > > a convention that is used by many file systems. It's a matter of
> > > > > familiarity with such path names that would help users in using
> HDFS
> > > > > snapshots.
> > > > >
> > > > > I am including the hdfs-dev in this email. Would appreciate if we
> > could
> > > > > work together and come up with a solution.
> > > > >
> > > > > You can find sample output from hdfs command here:
> > > > > http://pastebin.com/bBqR4Fvr
> > > > >
> > > > > Cheers
> > > >
> > > >
> > > >
> > > > --
> > > > // Jonathan Hsieh (shay)
> > > > // Software Engineer, Cloudera
> > > > // jon@cloudera.com
> > > >
> > >
> >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
I have a patch which touched these tests:
http://pastebin.com/P4p8LEAZ
I am running 0.94 test suite now - will publish patch and test result in
the morning.
Cheers
On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Alternatively, we can detect the underlying Hadoop version, and use
> either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>
> I think this would introduce more confusion, especially for operations.
>
> Cheers
>
> On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
>
>> Because HDFS exposes the snapshots so that the normal file system
>> operations are mapped inside snapshot dirs, I think HDFS reserving the
>> .snapshot name makes sense. OTOH, nothing is specific about the dir name
>> that is chosen by HBase.
>>
>> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
>> being run on top of hadoop 2. Alternatively, we can detect the underlying
>> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> depending on h1 & h2.
>>
>> Enis
>>
>>
>> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > bq. let's make the hbase snapshot for a conf variable.
>> >
>> > Once we decide on the new name of snapshot directory, we should still
>> use
>> > hardcoded value. This aligns with current code base:
>> > See this snippet from HConstants:
>> >
>> > public static final List<String> HBASE_NON_TABLE_DIRS =
>> >
>> > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > HREGION_LOGDIR_NAME,
>> >
>> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> >
>> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
>> > HBASE_TEMP_DIRECTORY }));
>> > Cheers
>> >
>> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
>> wrote:
>> >
>> > > constraints:
>> > >
>> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
>> > ever
>> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
>> current
>> > A
>> > > pache Hbase 0.94.x's.
>> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> should
>> > > pick a different table snapshot name to respect fs conventions.
>> > >
>> > > proposed actions:
>> > >
>> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > snapshots.dir) let's change the default for hbase 95+. (maybe
>> > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > 2) let's publish instructions on how to update the hbase snapshot dir:
>> > > shutdown hbase, config update, rename dir, restart hbase.
>> > > 3) I lean towards leaving the current default hbase snapshot dir in 94
>> > > since it shouldn't be affected. upgrading hbase to 95/96 will require
>> > > shutdown and update scripts so it seems like the ideal time to
>> autoforce
>> > > this default change.
>> > >
>> > > Thoughts?
>> > >
>> > >
>> > > On Monday, April 15, 2013, lars hofhansl wrote:
>> > >
>> > > > OK. Let's try to fix that quickly, so that I can release HBase
>> 0.94.7.
>> > > >
>> > > > -- Lars
>> > > >
>> > > >
>> > > >
>> > > > ________________________________
>> > > > From: Ted Yu <yu...@gmail.com>
>> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
>> > > > Sent: Monday, April 15, 2013 7:13 PM
>> > > > Subject: collision in the naming of '.snapshot' directory between
>> hdfs
>> > > > snapshot and hbase snapshot
>> > > >
>> > > >
>> > > > Hi,
>> > > > This afternoon Huned ad I discovered an issue while playing with
>> HBase
>> > > > Snapshots on top of Hadoop's Snapshot branch (
>> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>> > > >
>> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with
>> .snapshot
>> > as
>> > > a
>> > > > component while HBase tries to create paths with .snapshot as a
>> > > component.
>> > > > This leads to issues in HBase, and one of HDFS or HBase needs to
>> give
>> > up
>> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
>> > > 0.94.6
>> > > > (quite recently) and it may not be too late to change HBase to use a
>> > > > different path component in an upcoming new release.
>> > > >
>> > > > In HBase these path names are not user visible. If there is a
>> > deployment
>> > > of
>> > > > 0.94.6, one could provide a migration tool that renames .snapshot to
>> > > > .hbase-snapshot or something to be able to move to the Snapshot
>> release
>> > > of
>> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
>> and
>> > > is
>> > > > a convention that is used by many file systems. It's a matter of
>> > > > familiarity with such path names that would help users in using HDFS
>> > > > snapshots.
>> > > >
>> > > > I am including the hdfs-dev in this email. Would appreciate if we
>> could
>> > > > work together and come up with a solution.
>> > > >
>> > > > You can find sample output from hdfs command here:
>> > > > http://pastebin.com/bBqR4Fvr
>> > > >
>> > > > Cheers
>> > >
>> > >
>> > >
>> > > --
>> > > // Jonathan Hsieh (shay)
>> > > // Software Engineer, Cloudera
>> > > // jon@cloudera.com
>> > >
>> >
>>
>
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
bq. Alternatively, we can detect the underlying Hadoop version, and use
either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
I think this would introduce more confusion, especially for operations.
Cheers
On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
> Because HDFS exposes the snapshots so that the normal file system
> operations are mapped inside snapshot dirs, I think HDFS reserving the
> .snapshot name makes sense. OTOH, nothing is specific about the dir name
> that is chosen by HBase.
>
> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
> being run on top of hadoop 2. Alternatively, we can detect the underlying
> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> depending on h1 & h2.
>
> Enis
>
>
> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. let's make the hbase snapshot for a conf variable.
> >
> > Once we decide on the new name of snapshot directory, we should still use
> > hardcoded value. This aligns with current code base:
> > See this snippet from HConstants:
> >
> > public static final List<String> HBASE_NON_TABLE_DIRS =
> >
> > Collections.unmodifiableList(Arrays.asList(new String[] {
> > HREGION_LOGDIR_NAME,
> >
> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> >
> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
> > HBASE_TEMP_DIRECTORY }));
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > constraints:
> > >
> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
> > ever
> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> current
> > A
> > > pache Hbase 0.94.x's.
> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> should
> > > pick a different table snapshot name to respect fs conventions.
> > >
> > > proposed actions:
> > >
> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > 2) let's publish instructions on how to update the hbase snapshot dir:
> > > shutdown hbase, config update, rename dir, restart hbase.
> > > 3) I lean towards leaving the current default hbase snapshot dir in 94
> > > since it shouldn't be affected. upgrading hbase to 95/96 will require
> > > shutdown and update scripts so it seems like the ideal time to
> autoforce
> > > this default change.
> > >
> > > Thoughts?
> > >
> > >
> > > On Monday, April 15, 2013, lars hofhansl wrote:
> > >
> > > > OK. Let's try to fix that quickly, so that I can release HBase
> 0.94.7.
> > > >
> > > > -- Lars
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Ted Yu <yu...@gmail.com>
> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > Subject: collision in the naming of '.snapshot' directory between
> hdfs
> > > > snapshot and hbase snapshot
> > > >
> > > >
> > > > Hi,
> > > > This afternoon Huned ad I discovered an issue while playing with
> HBase
> > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > >
> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> > as
> > > a
> > > > component while HBase tries to create paths with .snapshot as a
> > > component.
> > > > This leads to issues in HBase, and one of HDFS or HBase needs to give
> > up
> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
> > > 0.94.6
> > > > (quite recently) and it may not be too late to change HBase to use a
> > > > different path component in an upcoming new release.
> > > >
> > > > In HBase these path names are not user visible. If there is a
> > deployment
> > > of
> > > > 0.94.6, one could provide a migration tool that renames .snapshot to
> > > > .hbase-snapshot or something to be able to move to the Snapshot
> release
> > > of
> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
> and
> > > is
> > > > a convention that is used by many file systems. It's a matter of
> > > > familiarity with such path names that would help users in using HDFS
> > > > snapshots.
> > > >
> > > > I am including the hdfs-dev in this email. Would appreciate if we
> could
> > > > work together and come up with a solution.
> > > >
> > > > You can find sample output from hdfs command here:
> > > > http://pastebin.com/bBqR4Fvr
> > > >
> > > > Cheers
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // Software Engineer, Cloudera
> > > // jon@cloudera.com
> > >
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
bq. Alternatively, we can detect the underlying Hadoop version, and use
either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
I think this would introduce more confusion, especially for operations.
Cheers
On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <en...@gmail.com> wrote:
> Because HDFS exposes the snapshots so that the normal file system
> operations are mapped inside snapshot dirs, I think HDFS reserving the
> .snapshot name makes sense. OTOH, nothing is specific about the dir name
> that is chosen by HBase.
>
> I would prefer to change the dir name in 0.94 as well, since 0.94 is also
> being run on top of hadoop 2. Alternatively, we can detect the underlying
> Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> depending on h1 & h2.
>
> Enis
>
>
> On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. let's make the hbase snapshot for a conf variable.
> >
> > Once we decide on the new name of snapshot directory, we should still use
> > hardcoded value. This aligns with current code base:
> > See this snippet from HConstants:
> >
> > public static final List<String> HBASE_NON_TABLE_DIRS =
> >
> > Collections.unmodifiableList(Arrays.asList(new String[] {
> > HREGION_LOGDIR_NAME,
> >
> > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> >
> > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
> > HBASE_TEMP_DIRECTORY }));
> > Cheers
> >
> > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > constraints:
> > >
> > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
> > ever
> > > make it to 1.x. This hdfs feature ideally this shouldn't affect
> current
> > A
> > > pache Hbase 0.94.x's.
> > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
> should
> > > pick a different table snapshot name to respect fs conventions.
> > >
> > > proposed actions:
> > >
> > > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > > snapshots.dir) let's change the default for hbase 95+. (maybe
> > > .hbase-snapshots). we'll also port this patch to 0.94.x
> > > 2) let's publish instructions on how to update the hbase snapshot dir:
> > > shutdown hbase, config update, rename dir, restart hbase.
> > > 3) I lean towards leaving the current default hbase snapshot dir in 94
> > > since it shouldn't be affected. upgrading hbase to 95/96 will require
> > > shutdown and update scripts so it seems like the ideal time to
> autoforce
> > > this default change.
> > >
> > > Thoughts?
> > >
> > >
> > > On Monday, April 15, 2013, lars hofhansl wrote:
> > >
> > > > OK. Let's try to fix that quickly, so that I can release HBase
> 0.94.7.
> > > >
> > > > -- Lars
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Ted Yu <yu...@gmail.com>
> > > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > > Sent: Monday, April 15, 2013 7:13 PM
> > > > Subject: collision in the naming of '.snapshot' directory between
> hdfs
> > > > snapshot and hbase snapshot
> > > >
> > > >
> > > > Hi,
> > > > This afternoon Huned ad I discovered an issue while playing with
> HBase
> > > > Snapshots on top of Hadoop's Snapshot branch (
> > > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > > >
> > > > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> > as
> > > a
> > > > component while HBase tries to create paths with .snapshot as a
> > > component.
> > > > This leads to issues in HBase, and one of HDFS or HBase needs to give
> > up
> > > > the .snapshot reserved keyword. HBase released Snapshots feature in
> > > 0.94.6
> > > > (quite recently) and it may not be too late to change HBase to use a
> > > > different path component in an upcoming new release.
> > > >
> > > > In HBase these path names are not user visible. If there is a
> > deployment
> > > of
> > > > 0.94.6, one could provide a migration tool that renames .snapshot to
> > > > .hbase-snapshot or something to be able to move to the Snapshot
> release
> > > of
> > > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name
> and
> > > is
> > > > a convention that is used by many file systems. It's a matter of
> > > > familiarity with such path names that would help users in using HDFS
> > > > snapshots.
> > > >
> > > > I am including the hdfs-dev in this email. Would appreciate if we
> could
> > > > work together and come up with a solution.
> > > >
> > > > You can find sample output from hdfs command here:
> > > > http://pastebin.com/bBqR4Fvr
> > > >
> > > > Cheers
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // Software Engineer, Cloudera
> > > // jon@cloudera.com
> > >
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Enis Söztutar <en...@gmail.com>.
Because HDFS exposes the snapshots so that the normal file system
operations are mapped inside snapshot dirs, I think HDFS reserving the
.snapshot name makes sense. OTOH, nothing is specific about the dir name
that is chosen by HBase.
I would prefer to change the dir name in 0.94 as well, since 0.94 is also
being run on top of hadoop 2. Alternatively, we can detect the underlying
Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
depending on h1 & h2.
Enis
On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. let's make the hbase snapshot for a conf variable.
>
> Once we decide on the new name of snapshot directory, we should still use
> hardcoded value. This aligns with current code base:
> See this snippet from HConstants:
>
> public static final List<String> HBASE_NON_TABLE_DIRS =
>
> Collections.unmodifiableList(Arrays.asList(new String[] {
> HREGION_LOGDIR_NAME,
>
> HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>
> HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
> HBASE_TEMP_DIRECTORY }));
> Cheers
>
> On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > constraints:
> >
> > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
> ever
> > make it to 1.x. This hdfs feature ideally this shouldn't affect current
> A
> > pache Hbase 0.94.x's.
> > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
> > pick a different table snapshot name to respect fs conventions.
> >
> > proposed actions:
> >
> > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > snapshots.dir) let's change the default for hbase 95+. (maybe
> > .hbase-snapshots). we'll also port this patch to 0.94.x
> > 2) let's publish instructions on how to update the hbase snapshot dir:
> > shutdown hbase, config update, rename dir, restart hbase.
> > 3) I lean towards leaving the current default hbase snapshot dir in 94
> > since it shouldn't be affected. upgrading hbase to 95/96 will require
> > shutdown and update scripts so it seems like the ideal time to autoforce
> > this default change.
> >
> > Thoughts?
> >
> >
> > On Monday, April 15, 2013, lars hofhansl wrote:
> >
> > > OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > Sent: Monday, April 15, 2013 7:13 PM
> > > Subject: collision in the naming of '.snapshot' directory between hdfs
> > > snapshot and hbase snapshot
> > >
> > >
> > > Hi,
> > > This afternoon Huned ad I discovered an issue while playing with HBase
> > > Snapshots on top of Hadoop's Snapshot branch (
> > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > >
> > > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> as
> > a
> > > component while HBase tries to create paths with .snapshot as a
> > component.
> > > This leads to issues in HBase, and one of HDFS or HBase needs to give
> up
> > > the .snapshot reserved keyword. HBase released Snapshots feature in
> > 0.94.6
> > > (quite recently) and it may not be too late to change HBase to use a
> > > different path component in an upcoming new release.
> > >
> > > In HBase these path names are not user visible. If there is a
> deployment
> > of
> > > 0.94.6, one could provide a migration tool that renames .snapshot to
> > > .hbase-snapshot or something to be able to move to the Snapshot release
> > of
> > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name and
> > is
> > > a convention that is used by many file systems. It's a matter of
> > > familiarity with such path names that would help users in using HDFS
> > > snapshots.
> > >
> > > I am including the hdfs-dev in this email. Would appreciate if we could
> > > work together and come up with a solution.
> > >
> > > You can find sample output from hdfs command here:
> > > http://pastebin.com/bBqR4Fvr
> > >
> > > Cheers
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Enis Söztutar <en...@gmail.com>.
Because HDFS exposes the snapshots so that the normal file system
operations are mapped inside snapshot dirs, I think HDFS reserving the
.snapshot name makes sense. OTOH, nothing is specific about the dir name
that is chosen by HBase.
I would prefer to change the dir name in 0.94 as well, since 0.94 is also
being run on top of hadoop 2. Alternatively, we can detect the underlying
Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
depending on h1 & h2.
Enis
On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. let's make the hbase snapshot for a conf variable.
>
> Once we decide on the new name of snapshot directory, we should still use
> hardcoded value. This aligns with current code base:
> See this snippet from HConstants:
>
> public static final List<String> HBASE_NON_TABLE_DIRS =
>
> Collections.unmodifiableList(Arrays.asList(new String[] {
> HREGION_LOGDIR_NAME,
>
> HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>
> HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
> HBASE_TEMP_DIRECTORY }));
> Cheers
>
> On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > constraints:
> >
> > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will
> ever
> > make it to 1.x. This hdfs feature ideally this shouldn't affect current
> A
> > pache Hbase 0.94.x's.
> > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
> > pick a different table snapshot name to respect fs conventions.
> >
> > proposed actions:
> >
> > 1) let's make the hbase snapshot for a conf variable. (hbase.
> > snapshots.dir) let's change the default for hbase 95+. (maybe
> > .hbase-snapshots). we'll also port this patch to 0.94.x
> > 2) let's publish instructions on how to update the hbase snapshot dir:
> > shutdown hbase, config update, rename dir, restart hbase.
> > 3) I lean towards leaving the current default hbase snapshot dir in 94
> > since it shouldn't be affected. upgrading hbase to 95/96 will require
> > shutdown and update scripts so it seems like the ideal time to autoforce
> > this default change.
> >
> > Thoughts?
> >
> >
> > On Monday, April 15, 2013, lars hofhansl wrote:
> >
> > > OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > > Sent: Monday, April 15, 2013 7:13 PM
> > > Subject: collision in the naming of '.snapshot' directory between hdfs
> > > snapshot and hbase snapshot
> > >
> > >
> > > Hi,
> > > This afternoon Huned ad I discovered an issue while playing with HBase
> > > Snapshots on top of Hadoop's Snapshot branch (
> > > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> > >
> > > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot
> as
> > a
> > > component while HBase tries to create paths with .snapshot as a
> > component.
> > > This leads to issues in HBase, and one of HDFS or HBase needs to give
> up
> > > the .snapshot reserved keyword. HBase released Snapshots feature in
> > 0.94.6
> > > (quite recently) and it may not be too late to change HBase to use a
> > > different path component in an upcoming new release.
> > >
> > > In HBase these path names are not user visible. If there is a
> deployment
> > of
> > > 0.94.6, one could provide a migration tool that renames .snapshot to
> > > .hbase-snapshot or something to be able to move to the Snapshot release
> > of
> > > Hadoop. On the other hand, .snapshot in HDFS is a user visible name and
> > is
> > > a convention that is used by many file systems. It's a matter of
> > > familiarity with such path names that would help users in using HDFS
> > > snapshots.
> > >
> > > I am including the hdfs-dev in this email. Would appreciate if we could
> > > work together and come up with a solution.
> > >
> > > You can find sample output from hdfs command here:
> > > http://pastebin.com/bBqR4Fvr
> > >
> > > Cheers
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
bq. let's make the hbase snapshot for a conf variable.
Once we decide on the new name of snapshot directory, we should still use
hardcoded value. This aligns with current code base:
See this snippet from HConstants:
public static final List<String> HBASE_NON_TABLE_DIRS =
Collections.unmodifiableList(Arrays.asList(new String[] {
HREGION_LOGDIR_NAME,
HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
HBASE_TEMP_DIRECTORY }));
Cheers
On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> constraints:
>
> 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever
> make it to 1.x. This hdfs feature ideally this shouldn't affect current A
> pache Hbase 0.94.x's.
> 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
> pick a different table snapshot name to respect fs conventions.
>
> proposed actions:
>
> 1) let's make the hbase snapshot for a conf variable. (hbase.
> snapshots.dir) let's change the default for hbase 95+. (maybe
> .hbase-snapshots). we'll also port this patch to 0.94.x
> 2) let's publish instructions on how to update the hbase snapshot dir:
> shutdown hbase, config update, rename dir, restart hbase.
> 3) I lean towards leaving the current default hbase snapshot dir in 94
> since it shouldn't be affected. upgrading hbase to 95/96 will require
> shutdown and update scripts so it seems like the ideal time to autoforce
> this default change.
>
> Thoughts?
>
>
> On Monday, April 15, 2013, lars hofhansl wrote:
>
> > OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > Sent: Monday, April 15, 2013 7:13 PM
> > Subject: collision in the naming of '.snapshot' directory between hdfs
> > snapshot and hbase snapshot
> >
> >
> > Hi,
> > This afternoon Huned ad I discovered an issue while playing with HBase
> > Snapshots on top of Hadoop's Snapshot branch (
> > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> >
> > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as
> a
> > component while HBase tries to create paths with .snapshot as a
> component.
> > This leads to issues in HBase, and one of HDFS or HBase needs to give up
> > the .snapshot reserved keyword. HBase released Snapshots feature in
> 0.94.6
> > (quite recently) and it may not be too late to change HBase to use a
> > different path component in an upcoming new release.
> >
> > In HBase these path names are not user visible. If there is a deployment
> of
> > 0.94.6, one could provide a migration tool that renames .snapshot to
> > .hbase-snapshot or something to be able to move to the Snapshot release
> of
> > Hadoop. On the other hand, .snapshot in HDFS is a user visible name and
> is
> > a convention that is used by many file systems. It's a matter of
> > familiarity with such path names that would help users in using HDFS
> > snapshots.
> >
> > I am including the hdfs-dev in this email. Would appreciate if we could
> > work together and come up with a solution.
> >
> > You can find sample output from hdfs command here:
> > http://pastebin.com/bBqR4Fvr
> >
> > Cheers
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
We can simplify this by just changing the snapshot dir name to .hbase_snapshot and publishing instructions about to rename the directory in case anybody used that feature in 0.94.6.
-- Lars
________________________________
From: Jonathan Hsieh <jo...@cloudera.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
Sent: Monday, April 15, 2013 8:24 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
constraints:
1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever
make it to 1.x. This hdfs feature ideally this shouldn't affect current A
pache Hbase 0.94.x's.
3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
pick a different table snapshot name to respect fs conventions.
proposed actions:
1) let's make the hbase snapshot for a conf variable. (hbase.
snapshots.dir) let's change the default for hbase 95+. (maybe
.hbase-snapshots). we'll also port this patch to 0.94.x
2) let's publish instructions on how to update the hbase snapshot dir:
shutdown hbase, config update, rename dir, restart hbase.
3) I lean towards leaving the current default hbase snapshot dir in 94
since it shouldn't be affected. upgrading hbase to 95/96 will require
shutdown and update scripts so it seems like the ideal time to autoforce
this default change.
Thoughts?
On Monday, April 15, 2013, lars hofhansl wrote:
> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>
> -- Lars
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
bq. let's make the hbase snapshot for a conf variable.
Once we decide on the new name of snapshot directory, we should still use
hardcoded value. This aligns with current code base:
See this snippet from HConstants:
public static final List<String> HBASE_NON_TABLE_DIRS =
Collections.unmodifiableList(Arrays.asList(new String[] {
HREGION_LOGDIR_NAME,
HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY, SNAPSHOT_DIR_NAME,
HBASE_TEMP_DIRECTORY }));
Cheers
On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> constraints:
>
> 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever
> make it to 1.x. This hdfs feature ideally this shouldn't affect current A
> pache Hbase 0.94.x's.
> 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
> pick a different table snapshot name to respect fs conventions.
>
> proposed actions:
>
> 1) let's make the hbase snapshot for a conf variable. (hbase.
> snapshots.dir) let's change the default for hbase 95+. (maybe
> .hbase-snapshots). we'll also port this patch to 0.94.x
> 2) let's publish instructions on how to update the hbase snapshot dir:
> shutdown hbase, config update, rename dir, restart hbase.
> 3) I lean towards leaving the current default hbase snapshot dir in 94
> since it shouldn't be affected. upgrading hbase to 95/96 will require
> shutdown and update scripts so it seems like the ideal time to autoforce
> this default change.
>
> Thoughts?
>
>
> On Monday, April 15, 2013, lars hofhansl wrote:
>
> > OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> > Sent: Monday, April 15, 2013 7:13 PM
> > Subject: collision in the naming of '.snapshot' directory between hdfs
> > snapshot and hbase snapshot
> >
> >
> > Hi,
> > This afternoon Huned ad I discovered an issue while playing with HBase
> > Snapshots on top of Hadoop's Snapshot branch (
> > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
> >
> > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as
> a
> > component while HBase tries to create paths with .snapshot as a
> component.
> > This leads to issues in HBase, and one of HDFS or HBase needs to give up
> > the .snapshot reserved keyword. HBase released Snapshots feature in
> 0.94.6
> > (quite recently) and it may not be too late to change HBase to use a
> > different path component in an upcoming new release.
> >
> > In HBase these path names are not user visible. If there is a deployment
> of
> > 0.94.6, one could provide a migration tool that renames .snapshot to
> > .hbase-snapshot or something to be able to move to the Snapshot release
> of
> > Hadoop. On the other hand, .snapshot in HDFS is a user visible name and
> is
> > a convention that is used by many file systems. It's a matter of
> > familiarity with such path names that would help users in using HDFS
> > snapshots.
> >
> > I am including the hdfs-dev in this email. Would appreciate if we could
> > work together and come up with a solution.
> >
> > You can find sample output from hdfs command here:
> > http://pastebin.com/bBqR4Fvr
> >
> > Cheers
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Jonathan Hsieh <jo...@cloudera.com>.
constraints:
1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever
make it to 1.x. This hdfs feature ideally this shouldn't affect current A
pache Hbase 0.94.x's.
3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should
pick a different table snapshot name to respect fs conventions.
proposed actions:
1) let's make the hbase snapshot for a conf variable. (hbase.
snapshots.dir) let's change the default for hbase 95+. (maybe
.hbase-snapshots). we'll also port this patch to 0.94.x
2) let's publish instructions on how to update the hbase snapshot dir:
shutdown hbase, config update, rename dir, restart hbase.
3) I lean towards leaving the current default hbase snapshot dir in 94
since it shouldn't be affected. upgrading hbase to 95/96 will require
shutdown and update scripts so it seems like the ideal time to autoforce
this default change.
Thoughts?
On Monday, April 15, 2013, lars hofhansl wrote:
> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>
> -- Lars
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Ted Yu <yu...@gmail.com>.
Lars:
I will go ahead and log an HBase JIRA, tomorrow morning.
This hopefully would give people enough time to respond.
Cheers
On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <la...@apache.org> wrote:
> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
>
> -- Lars
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
-- Lars
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
Sent: Monday, April 15, 2013 7:13 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Hi,
This afternoon Huned ad I discovered an issue while playing with HBase
Snapshots on top of Hadoop's Snapshot branch (
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
component while HBase tries to create paths with .snapshot as a component.
This leads to issues in HBase, and one of HDFS or HBase needs to give up
the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
(quite recently) and it may not be too late to change HBase to use a
different path component in an upcoming new release.
In HBase these path names are not user visible. If there is a deployment of
0.94.6, one could provide a migration tool that renames .snapshot to
.hbase-snapshot or something to be able to move to the Snapshot release of
Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
a convention that is used by many file systems. It's a matter of
familiarity with such path names that would help users in using HDFS
snapshots.
I am including the hdfs-dev in this email. Would appreciate if we could
work together and come up with a solution.
You can find sample output from hdfs command here:
http://pastebin.com/bBqR4Fvr
Cheers
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Azuryy Yu <az...@gmail.com>.
This was caused by HDFS-4666<https://issues.apache.org/jira/browse/HDFS-4666>,
committed yesterday.
IMO, it will be in a good place fianlly if HDFS and HBASE all changed
naming convention.
.hdfs_snapshot and .hbase_snapshot would be better.
On Tue, Apr 16, 2013 at 10:13 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
>
Re: collision in the naming of '.snapshot' directory between hdfs
snapshot and hbase snapshot
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
+1 for Jonathan's solution. We need to allow the user to configure that for
upcoming releases.
2013/4/15 Tsz Wo Sze <sz...@yahoo.com>
> Hi Ted,
>
> It would be great that if Hbase could use a different name since, as you
> mentioned, ".sanpshot" is user visible in HDFS. It is used for specifying
> snapshot paths. For example, the path /foo/.snapshot/s0/bar is used for
> accessing /foo/bar in snapshot s0 of /foo. Thanks.
>
>
> Tsz-Wo
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
> Sent: Monday, April 15, 2013 7:13 PM
> Subject: collision in the naming of '.snapshot' directory between hdfs
> snapshot and hbase snapshot
>
>
> Hi,
> This afternoon Huned ad I discovered an issue while playing with HBase
> Snapshots on top of Hadoop's Snapshot branch (
> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
>
> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
> component while HBase tries to create paths with .snapshot as a component.
> This leads to issues in HBase, and one of HDFS or HBase needs to give up
> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
> (quite recently) and it may not be too late to change HBase to use a
> different path component in an upcoming new release.
>
> In HBase these path names are not user visible. If there is a deployment of
> 0.94.6, one could provide a migration tool that renames .snapshot to
> .hbase-snapshot or something to be able to move to the Snapshot release of
> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
> a convention that is used by many file systems. It's a matter of
> familiarity with such path names that would help users in using HDFS
> snapshots.
>
> I am including the hdfs-dev in this email. Would appreciate if we could
> work together and come up with a solution.
>
> You can find sample output from hdfs command here:
> http://pastebin.com/bBqR4Fvr
>
> Cheers
>
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by Tsz Wo Sze <sz...@yahoo.com>.
Hi Ted,
It would be great that if Hbase could use a different name since, as you mentioned, ".sanpshot" is user visible in HDFS. It is used for specifying snapshot paths. For example, the path /foo/.snapshot/s0/bar is used for accessing /foo/bar in snapshot s0 of /foo. Thanks.
Tsz-Wo
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
Sent: Monday, April 15, 2013 7:13 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Hi,
This afternoon Huned ad I discovered an issue while playing with HBase
Snapshots on top of Hadoop's Snapshot branch (
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
component while HBase tries to create paths with .snapshot as a component.
This leads to issues in HBase, and one of HDFS or HBase needs to give up
the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
(quite recently) and it may not be too late to change HBase to use a
different path component in an upcoming new release.
In HBase these path names are not user visible. If there is a deployment of
0.94.6, one could provide a migration tool that renames .snapshot to
.hbase-snapshot or something to be able to move to the Snapshot release of
Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
a convention that is used by many file systems. It's a matter of
familiarity with such path names that would help users in using HDFS
snapshots.
I am including the hdfs-dev in this email. Would appreciate if we could
work together and come up with a solution.
You can find sample output from hdfs command here:
http://pastebin.com/bBqR4Fvr
Cheers
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by lars hofhansl <la...@apache.org>.
OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.
-- Lars
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
Sent: Monday, April 15, 2013 7:13 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Hi,
This afternoon Huned ad I discovered an issue while playing with HBase
Snapshots on top of Hadoop's Snapshot branch (
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
component while HBase tries to create paths with .snapshot as a component.
This leads to issues in HBase, and one of HDFS or HBase needs to give up
the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
(quite recently) and it may not be too late to change HBase to use a
different path component in an upcoming new release.
In HBase these path names are not user visible. If there is a deployment of
0.94.6, one could provide a migration tool that renames .snapshot to
.hbase-snapshot or something to be able to move to the Snapshot release of
Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
a convention that is used by many file systems. It's a matter of
familiarity with such path names that would help users in using HDFS
snapshots.
I am including the hdfs-dev in this email. Would appreciate if we could
work together and come up with a solution.
You can find sample output from hdfs command here:
http://pastebin.com/bBqR4Fvr
Cheers
Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Posted by Tsz Wo Sze <sz...@yahoo.com>.
Hi Ted,
It would be great that if Hbase could use a different name since, as you mentioned, ".sanpshot" is user visible in HDFS. It is used for specifying snapshot paths. For example, the path /foo/.snapshot/s0/bar is used for accessing /foo/bar in snapshot s0 of /foo. Thanks.
Tsz-Wo
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; hdfs-dev@hadoop.apache.org
Sent: Monday, April 15, 2013 7:13 PM
Subject: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot
Hi,
This afternoon Huned ad I discovered an issue while playing with HBase
Snapshots on top of Hadoop's Snapshot branch (
http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).
HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a
component while HBase tries to create paths with .snapshot as a component.
This leads to issues in HBase, and one of HDFS or HBase needs to give up
the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6
(quite recently) and it may not be too late to change HBase to use a
different path component in an upcoming new release.
In HBase these path names are not user visible. If there is a deployment of
0.94.6, one could provide a migration tool that renames .snapshot to
.hbase-snapshot or something to be able to move to the Snapshot release of
Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is
a convention that is used by many file systems. It's a matter of
familiarity with such path names that would help users in using HDFS
snapshots.
I am including the hdfs-dev in this email. Would appreciate if we could
work together and come up with a solution.
You can find sample output from hdfs command here:
http://pastebin.com/bBqR4Fvr
Cheers