You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by chutium <te...@gmail.com> on 2016/05/10 15:49:27 UTC
Re: Accumulo on s3
Hi Shawn, thanks for the detailed reply, we took a look into accumulo code,
and found the constant WAL_DIR defined in ServerConstants:
https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L139
https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L151
(we are using 1.7.1)
is that means, we can simply modify the return value of getWalDirs() to
change the location of WAL logs?
Thanks
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Accumulo-on-s3-tp16737p16872.html
Sent from the Developers mailing list archive at Nabble.com.
Re: Accumulo on s3
Posted by Christopher <ct...@apache.org>.
getWalDirs() is used only in legacy code intended to ease upgrades from
1.4.x to 1.5.x. It probably has no value in the current code, as I'm not
sure we've even tested upgrades from 1.4.x to 1.7.x.
On Thu, May 12, 2016 at 11:03 AM Shawn Walker <ac...@shawn-walker.net>
wrote:
> Teng Qiu,
>
> I don't believe your proposed change would suffice. I haven't spent much
> time exploring how one might implement my suggestion. Two things:
>
> 1. I've just recently started working with Accumulo; as such I'm not
> certain that WALs are the only files Accumulo uses that need more
> capability than S3 can provide. Just that it should be reasonable to store
> RFiles in S3, modulo some concerns about renaming.
>
> 2. Changing getWalDirs() would not be effective, as this function isn't
> used everywhere that matters. Indeed, my (quick) skim of the source
> indicates that getWalDirs() isn't used anywhere that matters. For example,
> the ultimate file location for a tserver's WAL looks to be decided at
> [tserver
> - DfsLogger.open(...):379]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java#L379
> >
> which doesn't call ServerConstants.getWalDirs().
>
> Instead, I was suggesting changing the signature of [server/base -
> VolumeManager.choose(...):161]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L161
> >
> to take an additional parameter indicating the purpose of the volume (RFile
> storage, WAL storage, temporary storage, etc), and passing that information
> along to the configured VolumeChooser by modifying [server/base -
> VolumeChooserEnvironment]
> <
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooserEnvironment.java
> >.
> The ultimate goal would be to allow a custom VolumeChooser to better choose
> volumes based on what type of use the volume would need to support.
>
> I'm not entirely certain this tactic would work, but it would be where I
> might start.
>
> --
> Shawn Walker
>
>
> On Tue, May 10, 2016 at 11:49 AM, chutium <te...@gmail.com> wrote:
>
> > Hi Shawn, thanks for the detailed reply, we took a look into accumulo
> code,
> > and found the constant WAL_DIR defined in ServerConstants:
> >
> >
> >
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L139
> >
> >
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L151
> > (we are using 1.7.1)
> >
> > is that means, we can simply modify the return value of getWalDirs() to
> > change the location of WAL logs?
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-accumulo.1065345.n5.nabble.com/Accumulo-on-s3-tp16737p16872.html
> > Sent from the Developers mailing list archive at Nabble.com.
> >
>
Re: Accumulo on s3
Posted by Shawn Walker <ac...@shawn-walker.net>.
Teng Qiu,
I don't believe your proposed change would suffice. I haven't spent much
time exploring how one might implement my suggestion. Two things:
1. I've just recently started working with Accumulo; as such I'm not
certain that WALs are the only files Accumulo uses that need more
capability than S3 can provide. Just that it should be reasonable to store
RFiles in S3, modulo some concerns about renaming.
2. Changing getWalDirs() would not be effective, as this function isn't
used everywhere that matters. Indeed, my (quick) skim of the source
indicates that getWalDirs() isn't used anywhere that matters. For example,
the ultimate file location for a tserver's WAL looks to be decided at [tserver
- DfsLogger.open(...):379]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/tserver/src/main/java/org/apache/accumulo/tserver/log/DfsLogger.java#L379>
which doesn't call ServerConstants.getWalDirs().
Instead, I was suggesting changing the signature of [server/base -
VolumeManager.choose(...):161]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeManager.java#L161>
to take an additional parameter indicating the purpose of the volume (RFile
storage, WAL storage, temporary storage, etc), and passing that information
along to the configured VolumeChooser by modifying [server/base -
VolumeChooserEnvironment]
<https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/fs/VolumeChooserEnvironment.java>.
The ultimate goal would be to allow a custom VolumeChooser to better choose
volumes based on what type of use the volume would need to support.
I'm not entirely certain this tactic would work, but it would be where I
might start.
--
Shawn Walker
On Tue, May 10, 2016 at 11:49 AM, chutium <te...@gmail.com> wrote:
> Hi Shawn, thanks for the detailed reply, we took a look into accumulo code,
> and found the constant WAL_DIR defined in ServerConstants:
>
>
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L139
>
> https://github.com/apache/accumulo/blob/rel/1.7.1/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java#L151
> (we are using 1.7.1)
>
> is that means, we can simply modify the return value of getWalDirs() to
> change the location of WAL logs?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/Accumulo-on-s3-tp16737p16872.html
> Sent from the Developers mailing list archive at Nabble.com.
>