You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Sven Hodapp <sv...@scai.fraunhofer.de> on 2015/09/16 15:20:52 UTC
Mini Accumulo Cluster reusing the directory
Hi there,
is it possible for MiniAccumuloCluster to reuse a given directory?
Sadly, I haven't found anything in the docs?
I’ll fire up my instance like this:
val dict = new File("/tmp/accumulo-mini-cluster")
val accumulo = new MiniAccumuloCluster(dict, "test“)
If I’ll restart my JVM it will raise a error like this:
Exception in thread "main" java.lang.IllegalArgumentException: Directory /tmp/accumulo-mini-cluster is not empty
It would be nice if the data can survive a JVM restart and the folder structure must not be constructed every time.
Thanks a lot!
Regards,
Sven
--
Sven Hodapp M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
Re: Mini Accumulo Cluster reusing the directory
Posted by Josh Elser <jo...@gmail.com>.
n/m, didn't read the whole thread. Sounds like ACCUMULO-1378 does
capture what you asked for :)
Josh Elser wrote:
> Best as I can tell, we don't actually have the ability to do this.
>
> We can use external ZooKeeper servers and we can start a MAC instance on
> top of an already running ZK from a different MAC instance, but it
> doesn't look like we have the ability to reuse the same storage directory.
>
> That sounds like a useful improvement to me. I'm not sure if there are
> any edge cases that make this hard (haven't thought enough about it).
>
> Want to open a JIRA issue for it, Sven?
>
> - Josh
>
> Sven Hodapp wrote:
>> Hi there,
>>
>> is it possible for MiniAccumuloCluster to reuse a given directory?
>> Sadly, I haven't found anything in the docs?
>>
>> I’ll fire up my instance like this:
>>
>> val dict = new File("/tmp/accumulo-mini-cluster")
>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>
>> If I’ll restart my JVM it will raise a error like this:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> Directory /tmp/accumulo-mini-cluster is not empty
>>
>> It would be nice if the data can survive a JVM restart and the folder
>> structure must not be constructed every time.
>>
>> Thanks a lot!
>>
>> Regards,
>> Sven
>>
Re: Mini Accumulo Cluster reusing the directory
Posted by Josh Elser <jo...@gmail.com>.
Best as I can tell, we don't actually have the ability to do this.
We can use external ZooKeeper servers and we can start a MAC instance on
top of an already running ZK from a different MAC instance, but it
doesn't look like we have the ability to reuse the same storage directory.
That sounds like a useful improvement to me. I'm not sure if there are
any edge cases that make this hard (haven't thought enough about it).
Want to open a JIRA issue for it, Sven?
- Josh
Sven Hodapp wrote:
> Hi there,
>
> is it possible for MiniAccumuloCluster to reuse a given directory?
> Sadly, I haven't found anything in the docs?
>
> I’ll fire up my instance like this:
>
> val dict = new File("/tmp/accumulo-mini-cluster")
> val accumulo = new MiniAccumuloCluster(dict, "test“)
>
> If I’ll restart my JVM it will raise a error like this:
>
> Exception in thread "main" java.lang.IllegalArgumentException: Directory /tmp/accumulo-mini-cluster is not empty
>
> It would be nice if the data can survive a JVM restart and the folder structure must not be constructed every time.
>
> Thanks a lot!
>
> Regards,
> Sven
>
Re: Mini Accumulo Cluster reusing the directory
Posted by "mohit.kaushik" <mo...@orkash.com>.
Thanks Josh :)
On 09/17/2015 10:09 AM, Josh Elser wrote:
> And, to be explicit, if you want to configure not-MiniAccumuloCluster
> to use the local FS (you only run on one node), just use file:///...
> in instance.volumes (or instance.dfs.uri) instead of
> hdfs://namenode:port/...
>
> Wasn't sure if it would be implied to look at how MAC does this from
> my previous response.
>
> Josh Elser wrote:
>> MiniAccumuloCluster by default already uses the local file system.
>>
>> To get the correct sync semantics that Keith mentioned you need to set
>> "fs.file.impl" in the Hadoop Configuration you pass to the
>> MiniAccumuloConfig.
>>
>> See
>> https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/VolumeIT.java#L113
>>
>> for code.
>>
>> mohit.kaushik wrote:
>>> Keith,
>>>
>>> How can I configure Accumulo to use local file system???
>>>
>>> On 09/17/2015 01:11 AM, Keith Turner wrote:
>>>> Would you be able to provide more informaiton about your use case?
>>>> Was wondering if other solutions could be of use, like configuring
>>>> regular Accumulo to use the local filesystem. This can be done, but
>>>> care needs to be taken to make walogs work correctly. If interested
>>>> I could provide more info about this configuration.
>>>>
>>>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
>>>> <sven.hodapp@scai.fraunhofer.de
>>>> <ma...@scai.fraunhofer.de>> wrote:
>>>>
>>>> Hi there,
>>>>
>>>> is it possible for MiniAccumuloCluster to reuse a given directory?
>>>> Sadly, I haven't found anything in the docs?
>>>>
>>>> I’ll fire up my instance like this:
>>>>
>>>> val dict = new File("/tmp/accumulo-mini-cluster")
>>>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>>>
>>>> If I’ll restart my JVM it will raise a error like this:
>>>>
>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> Directory /tmp/accumulo-mini-cluster is not empty
>>>>
>>>> It would be nice if the data can survive a JVM restart and the
>>>> folder structure must not be constructed every time.
>>>>
>>>> Thanks a lot!
>>>>
>>>> Regards,
>>>> Sven
>>>>
>>>> --
>>>> Sven Hodapp M.Sc.,
>>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>>> Department of Bioinformatics
>>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>>> sven.hodapp@scai.fraunhofer.de <ma...@scai.fraunhofer.de>
>>>> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *Mohit Kaushik*
>>> Software Engineer
>>> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
>>> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>>>
>>> <http://politicomapper.orkash.com>interactive social intelligence at
>>> work...
>>>
>>> <https://www.facebook.com/Orkash2012>
>>> <http://www.linkedin.com/company/orkash-services-private-limited>
>>> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
>>> <http://www.orkash.com>
>>> <http://www.orkash.com> ... ensuring Assurance in complexity and
>>> uncertainty
>>>
>>> /This message including the attachments, if any, is a confidential
>>> business communication. If you are not the intended recipient it may be
>>> unlawful for you to read, copy, distribute, disclose or otherwise use
>>> the information in this e-mail. If you have received it in error or are
>>> not the intended recipient, please destroy it and notify the sender
>>> immediately. Thank you /
>>>
>
>
--
Signature
*Mohit Kaushik*
Software Engineer
A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
*Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
<http://politicomapper.orkash.com>interactive social intelligence at work...
<https://www.facebook.com/Orkash2012>
<http://www.linkedin.com/company/orkash-services-private-limited>
<https://twitter.com/Orkash> <http://www.orkash.com/blog/>
<http://www.orkash.com>
<http://www.orkash.com> ... ensuring Assurance in complexity and uncertainty
/This message including the attachments, if any, is a confidential
business communication. If you are not the intended recipient it may be
unlawful for you to read, copy, distribute, disclose or otherwise use
the information in this e-mail. If you have received it in error or are
not the intended recipient, please destroy it and notify the sender
immediately. Thank you /
Re: Mini Accumulo Cluster reusing the directory
Posted by Josh Elser <jo...@gmail.com>.
And, to be explicit, if you want to configure not-MiniAccumuloCluster to
use the local FS (you only run on one node), just use file:///... in
instance.volumes (or instance.dfs.uri) instead of hdfs://namenode:port/...
Wasn't sure if it would be implied to look at how MAC does this from my
previous response.
Josh Elser wrote:
> MiniAccumuloCluster by default already uses the local file system.
>
> To get the correct sync semantics that Keith mentioned you need to set
> "fs.file.impl" in the Hadoop Configuration you pass to the
> MiniAccumuloConfig.
>
> See
> https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/VolumeIT.java#L113
> for code.
>
> mohit.kaushik wrote:
>> Keith,
>>
>> How can I configure Accumulo to use local file system???
>>
>> On 09/17/2015 01:11 AM, Keith Turner wrote:
>>> Would you be able to provide more informaiton about your use case?
>>> Was wondering if other solutions could be of use, like configuring
>>> regular Accumulo to use the local filesystem. This can be done, but
>>> care needs to be taken to make walogs work correctly. If interested
>>> I could provide more info about this configuration.
>>>
>>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
>>> <sven.hodapp@scai.fraunhofer.de
>>> <ma...@scai.fraunhofer.de>> wrote:
>>>
>>> Hi there,
>>>
>>> is it possible for MiniAccumuloCluster to reuse a given directory?
>>> Sadly, I haven't found anything in the docs?
>>>
>>> I’ll fire up my instance like this:
>>>
>>> val dict = new File("/tmp/accumulo-mini-cluster")
>>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>>
>>> If I’ll restart my JVM it will raise a error like this:
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> Directory /tmp/accumulo-mini-cluster is not empty
>>>
>>> It would be nice if the data can survive a JVM restart and the
>>> folder structure must not be constructed every time.
>>>
>>> Thanks a lot!
>>>
>>> Regards,
>>> Sven
>>>
>>> --
>>> Sven Hodapp M.Sc.,
>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> Department of Bioinformatics
>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> sven.hodapp@scai.fraunhofer.de <ma...@scai.fraunhofer.de>
>>> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>>>
>>>
>>
>>
>> --
>>
>> *Mohit Kaushik*
>> Software Engineer
>> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
>> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>>
>> <http://politicomapper.orkash.com>interactive social intelligence at
>> work...
>>
>> <https://www.facebook.com/Orkash2012>
>> <http://www.linkedin.com/company/orkash-services-private-limited>
>> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
>> <http://www.orkash.com>
>> <http://www.orkash.com> ... ensuring Assurance in complexity and
>> uncertainty
>>
>> /This message including the attachments, if any, is a confidential
>> business communication. If you are not the intended recipient it may be
>> unlawful for you to read, copy, distribute, disclose or otherwise use
>> the information in this e-mail. If you have received it in error or are
>> not the intended recipient, please destroy it and notify the sender
>> immediately. Thank you /
>>
Re: Mini Accumulo Cluster reusing the directory
Posted by Keith Turner <ke...@deenlo.com>.
On Thu, Sep 17, 2015 at 12:38 AM, Josh Elser <jo...@gmail.com> wrote:
> MiniAccumuloCluster by default already uses the local file system.
>
> To get the correct sync semantics that Keith mentioned you need to set
> "fs.file.impl" in the Hadoop Configuration you pass to the
> MiniAccumuloConfig.
>
Yeah thats the main thing I was thinking of. Thanks for digging it up
Josh. Its really important to set that hadoop config when having Accumulo
use the local FS. Otherwise the default class hadoop uses for the local fs
does nothing for flush.
A few caveats to be aware of :
* The default impl is LocalFileSystem and it used to do nothing for
flush(). That may no longer be true, at one point it extended something
from CheckSumFS which did nothing for flush.
* RawLocalFileSystem may not checksum data like LocalFileSystem does
* RawLocalFileSystem may not survive a power outage. It flushes the
file to the OS... but the OS may hold it in its buffers for some time
before really writing it to disk. So this means you would survive
processes terminating, but not the OS terminating unexpectedly. When
using HDFS Accumulo can be configured to make the hdfs output stream make
java.nio calls to flush data to disk.
> See
> https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/VolumeIT.java#L113
> for code.
>
> mohit.kaushik wrote:
>
>> Keith,
>>
>> How can I configure Accumulo to use local file system???
>>
>> On 09/17/2015 01:11 AM, Keith Turner wrote:
>>
>>> Would you be able to provide more informaiton about your use case?
>>> Was wondering if other solutions could be of use, like configuring
>>> regular Accumulo to use the local filesystem. This can be done, but
>>> care needs to be taken to make walogs work correctly. If interested
>>> I could provide more info about this configuration.
>>>
>>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
>>> <sven.hodapp@scai.fraunhofer.de
>>> <ma...@scai.fraunhofer.de>> wrote:
>>>
>>> Hi there,
>>>
>>> is it possible for MiniAccumuloCluster to reuse a given directory?
>>> Sadly, I haven't found anything in the docs?
>>>
>>> I’ll fire up my instance like this:
>>>
>>> val dict = new File("/tmp/accumulo-mini-cluster")
>>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>>
>>> If I’ll restart my JVM it will raise a error like this:
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> Directory /tmp/accumulo-mini-cluster is not empty
>>>
>>> It would be nice if the data can survive a JVM restart and the
>>> folder structure must not be constructed every time.
>>>
>>> Thanks a lot!
>>>
>>> Regards,
>>> Sven
>>>
>>> --
>>> Sven Hodapp M.Sc.,
>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> Department of Bioinformatics
>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> sven.hodapp@scai.fraunhofer.de <mailto:
>>> sven.hodapp@scai.fraunhofer.de>
>>> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>>>
>>>
>>>
>>
>> --
>>
>> *Mohit Kaushik*
>> Software Engineer
>> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
>> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>>
>> <http://politicomapper.orkash.com>interactive social intelligence at
>> work...
>>
>> <https://www.facebook.com/Orkash2012>
>> <http://www.linkedin.com/company/orkash-services-private-limited>
>> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
>> <http://www.orkash.com>
>> <http://www.orkash.com> ... ensuring Assurance in complexity and
>> uncertainty
>>
>> /This message including the attachments, if any, is a confidential
>> business communication. If you are not the intended recipient it may be
>> unlawful for you to read, copy, distribute, disclose or otherwise use
>> the information in this e-mail. If you have received it in error or are
>> not the intended recipient, please destroy it and notify the sender
>> immediately. Thank you /
>>
>>
Re: Mini Accumulo Cluster reusing the directory
Posted by Josh Elser <jo...@gmail.com>.
MiniAccumuloCluster by default already uses the local file system.
To get the correct sync semantics that Keith mentioned you need to set
"fs.file.impl" in the Hadoop Configuration you pass to the
MiniAccumuloConfig.
See
https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/VolumeIT.java#L113
for code.
mohit.kaushik wrote:
> Keith,
>
> How can I configure Accumulo to use local file system???
>
> On 09/17/2015 01:11 AM, Keith Turner wrote:
>> Would you be able to provide more informaiton about your use case?
>> Was wondering if other solutions could be of use, like configuring
>> regular Accumulo to use the local filesystem. This can be done, but
>> care needs to be taken to make walogs work correctly. If interested
>> I could provide more info about this configuration.
>>
>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
>> <sven.hodapp@scai.fraunhofer.de
>> <ma...@scai.fraunhofer.de>> wrote:
>>
>> Hi there,
>>
>> is it possible for MiniAccumuloCluster to reuse a given directory?
>> Sadly, I haven't found anything in the docs?
>>
>> I’ll fire up my instance like this:
>>
>> val dict = new File("/tmp/accumulo-mini-cluster")
>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>
>> If I’ll restart my JVM it will raise a error like this:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> Directory /tmp/accumulo-mini-cluster is not empty
>>
>> It would be nice if the data can survive a JVM restart and the
>> folder structure must not be constructed every time.
>>
>> Thanks a lot!
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de <ma...@scai.fraunhofer.de>
>> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>>
>>
>
>
> --
>
> *Mohit Kaushik*
> Software Engineer
> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>
> <http://politicomapper.orkash.com>interactive social intelligence at work...
>
> <https://www.facebook.com/Orkash2012>
> <http://www.linkedin.com/company/orkash-services-private-limited>
> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
> <http://www.orkash.com>
> <http://www.orkash.com> ... ensuring Assurance in complexity and uncertainty
>
> /This message including the attachments, if any, is a confidential
> business communication. If you are not the intended recipient it may be
> unlawful for you to read, copy, distribute, disclose or otherwise use
> the information in this e-mail. If you have received it in error or are
> not the intended recipient, please destroy it and notify the sender
> immediately. Thank you /
>
Re: Mini Accumulo Cluster reusing the directory
Posted by "mohit.kaushik" <mo...@orkash.com>.
Keith,
How can I configure Accumulo to use local file system???
On 09/17/2015 01:11 AM, Keith Turner wrote:
> Would you be able to provide more informaiton about your use case?
> Was wondering if other solutions could be of use, like configuring
> regular Accumulo to use the local filesystem. This can be done, but
> care needs to be taken to make walogs work correctly. If interested
> I could provide more info about this configuration.
>
> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
> <sven.hodapp@scai.fraunhofer.de
> <ma...@scai.fraunhofer.de>> wrote:
>
> Hi there,
>
> is it possible for MiniAccumuloCluster to reuse a given directory?
> Sadly, I haven't found anything in the docs?
>
> I’ll fire up my instance like this:
>
> val dict = new File("/tmp/accumulo-mini-cluster")
> val accumulo = new MiniAccumuloCluster(dict, "test“)
>
> If I’ll restart my JVM it will raise a error like this:
>
> Exception in thread "main" java.lang.IllegalArgumentException:
> Directory /tmp/accumulo-mini-cluster is not empty
>
> It would be nice if the data can survive a JVM restart and the
> folder structure must not be constructed every time.
>
> Thanks a lot!
>
> Regards,
> Sven
>
> --
> Sven Hodapp M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de <ma...@scai.fraunhofer.de>
> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>
>
--
Signature
*Mohit Kaushik*
Software Engineer
A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
*Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
<http://politicomapper.orkash.com>interactive social intelligence at work...
<https://www.facebook.com/Orkash2012>
<http://www.linkedin.com/company/orkash-services-private-limited>
<https://twitter.com/Orkash> <http://www.orkash.com/blog/>
<http://www.orkash.com>
<http://www.orkash.com> ... ensuring Assurance in complexity and uncertainty
/This message including the attachments, if any, is a confidential
business communication. If you are not the intended recipient it may be
unlawful for you to read, copy, distribute, disclose or otherwise use
the information in this e-mail. If you have received it in error or are
not the intended recipient, please destroy it and notify the sender
immediately. Thank you /
Re: Mini Accumulo Cluster reusing the directory
Posted by Andrew Wells <aw...@clearedgeit.com>.
The last is a cool idea... especially if you scaled out from that
application install. Using MAC for that would mean there would be some
downtime.
On Tue, Sep 22, 2015 at 3:42 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
> wrote:
> Hi Keith,
>
> for me the use cases are:
>
> * easy and portable development
> * in general testing
> * embedding Accumulo
>
> Especially the last point is for me very insteresting.
> First I can deliver a relativly light weight application, which has
> Accumulo embedded (like a library).
> And then, if the application runs very well and gets many users, it's very
> easy to scale to a full Accumulo installation!
>
> Regards,
> Sven
>
> --
> Sven Hodapp M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>
> ----- Ursprüngliche Mail -----
> > Von: "Keith Turner" <ke...@deenlo.com>
> > An: "user" <us...@accumulo.apache.org>
> > Gesendet: Mittwoch, 16. September 2015 21:41:00
> > Betreff: Re: Mini Accumulo Cluster reusing the directory
>
> > Would you be able to provide more informaiton about your use case? Was
> > wondering if other solutions could be of use, like configuring regular
> > Accumulo to use the local filesystem. This can be done, but care needs
> to
> > be taken to make walogs work correctly. If interested I could provide
> > more info about this configuration.
> >
> > On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
> sven.hodapp@scai.fraunhofer.de
> >> wrote:
> >
> >> Hi there,
> >>
> >> is it possible for MiniAccumuloCluster to reuse a given directory?
> >> Sadly, I haven't found anything in the docs?
> >>
> >> I’ll fire up my instance like this:
> >>
> >> val dict = new File("/tmp/accumulo-mini-cluster")
> >> val accumulo = new MiniAccumuloCluster(dict, "test“)
> >>
> >> If I’ll restart my JVM it will raise a error like this:
> >>
> >> Exception in thread "main" java.lang.IllegalArgumentException:
> >> Directory /tmp/accumulo-mini-cluster is not empty
> >>
> >> It would be nice if the data can survive a JVM restart and the folder
> >> structure must not be constructed every time.
> >>
> >> Thanks a lot!
> >>
> >> Regards,
> >> Sven
> >>
> >> --
> >> Sven Hodapp M.Sc.,
> >> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> >> Department of Bioinformatics
> >> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> >> sven.hodapp@scai.fraunhofer.de
> >> www.scai.fraunhofer.de
>
--
*Andrew George Wells*
*Software Engineer*
*awells@clearedgeit.com <aw...@clearedgeit.com>*
Re: Mini Accumulo Cluster reusing the directory
Posted by Sven Hodapp <sv...@scai.fraunhofer.de>.
Hi Keith,
for me the use cases are:
* easy and portable development
* in general testing
* embedding Accumulo
Especially the last point is for me very insteresting.
First I can deliver a relativly light weight application, which has Accumulo embedded (like a library).
And then, if the application runs very well and gets many users, it's very easy to scale to a full Accumulo installation!
Regards,
Sven
--
Sven Hodapp M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
----- Ursprüngliche Mail -----
> Von: "Keith Turner" <ke...@deenlo.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Mittwoch, 16. September 2015 21:41:00
> Betreff: Re: Mini Accumulo Cluster reusing the directory
> Would you be able to provide more informaiton about your use case? Was
> wondering if other solutions could be of use, like configuring regular
> Accumulo to use the local filesystem. This can be done, but care needs to
> be taken to make walogs work correctly. If interested I could provide
> more info about this configuration.
>
> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
>> wrote:
>
>> Hi there,
>>
>> is it possible for MiniAccumuloCluster to reuse a given directory?
>> Sadly, I haven't found anything in the docs?
>>
>> I’ll fire up my instance like this:
>>
>> val dict = new File("/tmp/accumulo-mini-cluster")
>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>
>> If I’ll restart my JVM it will raise a error like this:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> Directory /tmp/accumulo-mini-cluster is not empty
>>
>> It would be nice if the data can survive a JVM restart and the folder
>> structure must not be constructed every time.
>>
>> Thanks a lot!
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de
>> www.scai.fraunhofer.de
Re: Mini Accumulo Cluster reusing the directory
Posted by Keith Turner <ke...@deenlo.com>.
Would you be able to provide more informaiton about your use case? Was
wondering if other solutions could be of use, like configuring regular
Accumulo to use the local filesystem. This can be done, but care needs to
be taken to make walogs work correctly. If interested I could provide
more info about this configuration.
On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
> wrote:
> Hi there,
>
> is it possible for MiniAccumuloCluster to reuse a given directory?
> Sadly, I haven't found anything in the docs?
>
> I’ll fire up my instance like this:
>
> val dict = new File("/tmp/accumulo-mini-cluster")
> val accumulo = new MiniAccumuloCluster(dict, "test“)
>
> If I’ll restart my JVM it will raise a error like this:
>
> Exception in thread "main" java.lang.IllegalArgumentException:
> Directory /tmp/accumulo-mini-cluster is not empty
>
> It would be nice if the data can survive a JVM restart and the folder
> structure must not be constructed every time.
>
> Thanks a lot!
>
> Regards,
> Sven
>
> --
> Sven Hodapp M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>
Re: Mini Accumulo Cluster reusing the directory
Posted by Andrew Wells <aw...@clearedgeit.com>.
btw, my solution revolved around this issue:
https://issues.apache.org/jira/browse/ACCUMULO-2851
which let users save their data off somewhere, then load it into the new
mac cluster
On Wed, Sep 16, 2015 at 10:37 AM, Andrew Wells <aw...@clearedgeit.com>
wrote:
> I created a util for doing this, but i haven't worked on it in a while...
>
> You could see if it still works
>
> https://github.com/agwells0714/AccumuloDeveloperUtil
>
> On Wed, Sep 16, 2015 at 10:31 AM, Corey Nolet <cj...@gmail.com> wrote:
>
>> Sven,
>>
>> What version of Accumulo are you running? We have a ticket for this [1]
>> which has had a lot of discussion on it. Christopher Tubbs mentioned that
>> he had gotten this to work.
>>
>> [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
>>
>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
>> sven.hodapp@scai.fraunhofer.de> wrote:
>>
>>> Hi there,
>>>
>>> is it possible for MiniAccumuloCluster to reuse a given directory?
>>> Sadly, I haven't found anything in the docs?
>>>
>>> I’ll fire up my instance like this:
>>>
>>> val dict = new File("/tmp/accumulo-mini-cluster")
>>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>>
>>> If I’ll restart my JVM it will raise a error like this:
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> Directory /tmp/accumulo-mini-cluster is not empty
>>>
>>> It would be nice if the data can survive a JVM restart and the folder
>>> structure must not be constructed every time.
>>>
>>> Thanks a lot!
>>>
>>> Regards,
>>> Sven
>>>
>>> --
>>> Sven Hodapp M.Sc.,
>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> Department of Bioinformatics
>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> sven.hodapp@scai.fraunhofer.de
>>> www.scai.fraunhofer.de
>>>
>>
>>
>
>
> --
> *Andrew George Wells*
> *Software Engineer*
> *awells@clearedgeit.com <aw...@clearedgeit.com>*
>
>
--
*Andrew George Wells*
*Software Engineer*
*awells@clearedgeit.com <aw...@clearedgeit.com>*
Re: Mini Accumulo Cluster reusing the directory
Posted by Andrew Wells <aw...@clearedgeit.com>.
I created a util for doing this, but i haven't worked on it in a while...
You could see if it still works
https://github.com/agwells0714/AccumuloDeveloperUtil
On Wed, Sep 16, 2015 at 10:31 AM, Corey Nolet <cj...@gmail.com> wrote:
> Sven,
>
> What version of Accumulo are you running? We have a ticket for this [1]
> which has had a lot of discussion on it. Christopher Tubbs mentioned that
> he had gotten this to work.
>
> [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
>
> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
> sven.hodapp@scai.fraunhofer.de> wrote:
>
>> Hi there,
>>
>> is it possible for MiniAccumuloCluster to reuse a given directory?
>> Sadly, I haven't found anything in the docs?
>>
>> I’ll fire up my instance like this:
>>
>> val dict = new File("/tmp/accumulo-mini-cluster")
>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>
>> If I’ll restart my JVM it will raise a error like this:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> Directory /tmp/accumulo-mini-cluster is not empty
>>
>> It would be nice if the data can survive a JVM restart and the folder
>> structure must not be constructed every time.
>>
>> Thanks a lot!
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de
>> www.scai.fraunhofer.de
>>
>
>
--
*Andrew George Wells*
*Software Engineer*
*awells@clearedgeit.com <aw...@clearedgeit.com>*
Re: Mini Accumulo Cluster reusing the directory
Posted by Christopher <ct...@apache.org>.
The class I created is included in Accumulo (since 1.6?), but only enables
connecting to an existing mini instance based on its directory. It doesn't
enable restarting mini.
On Wed, Sep 16, 2015, 11:40 Corey Nolet <cj...@gmail.com> wrote:
> Christopher, my reply to Sven was just going off of your last comment on
> the ticket. Perhaps I misread the comment.
>
> On Wed, Sep 16, 2015 at 10:50 AM, Christopher <ct...@apache.org> wrote:
>
>> Hi Sven, I think Corey was mistaken. I don't recall working on that,
>> exactly.
>>
>> On Wed, Sep 16, 2015, 10:40 Sven Hodapp <sv...@scai.fraunhofer.de>
>> wrote:
>>
>>> Hi Corey,
>>>
>>> thanks for your reply and the link. Sounds good, if that will be
>>> available in the future!
>>> Is the code from Christopher somewhere deployed?
>>>
>>> Currently I'm using version 1.7
>>>
>>> Regards,
>>> Sven
>>>
>>> ----- Ursprüngliche Mail -----
>>> > Von: "Corey Nolet" <cj...@gmail.com>
>>> > An: "user" <us...@accumulo.apache.org>
>>> > Gesendet: Mittwoch, 16. September 2015 16:31:02
>>> > Betreff: Re: Mini Accumulo Cluster reusing the directory
>>>
>>> > Sven,
>>> >
>>> > What version of Accumulo are you running? We have a ticket for this [1]
>>> > which has had a lot of discussion on it. Christopher Tubbs mentioned
>>> that
>>> > he had gotten this to work.
>>> >
>>> > [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
>>> >
>>> > On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
>>> sven.hodapp@scai.fraunhofer.de
>>> >> wrote:
>>> >
>>> >> Hi there,
>>> >>
>>> >> is it possible for MiniAccumuloCluster to reuse a given directory?
>>> >> Sadly, I haven't found anything in the docs?
>>> >>
>>> >> I’ll fire up my instance like this:
>>> >>
>>> >> val dict = new File("/tmp/accumulo-mini-cluster")
>>> >> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>> >>
>>> >> If I’ll restart my JVM it will raise a error like this:
>>> >>
>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>> >> Directory /tmp/accumulo-mini-cluster is not empty
>>> >>
>>> >> It would be nice if the data can survive a JVM restart and the folder
>>> >> structure must not be constructed every time.
>>> >>
>>> >> Thanks a lot!
>>> >>
>>> >> Regards,
>>> >> Sven
>>> >>
>>> >> --
>>> >> Sven Hodapp M.Sc.,
>>> >> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> >> Department of Bioinformatics
>>> >> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> >> sven.hodapp@scai.fraunhofer.de
>>> >> www.scai.fraunhofer.de
>>>
>>
>
Re: Mini Accumulo Cluster reusing the directory
Posted by Corey Nolet <cj...@gmail.com>.
Christopher, my reply to Sven was just going off of your last comment on
the ticket. Perhaps I misread the comment.
On Wed, Sep 16, 2015 at 10:50 AM, Christopher <ct...@apache.org> wrote:
> Hi Sven, I think Corey was mistaken. I don't recall working on that,
> exactly.
>
> On Wed, Sep 16, 2015, 10:40 Sven Hodapp <sv...@scai.fraunhofer.de>
> wrote:
>
>> Hi Corey,
>>
>> thanks for your reply and the link. Sounds good, if that will be
>> available in the future!
>> Is the code from Christopher somewhere deployed?
>>
>> Currently I'm using version 1.7
>>
>> Regards,
>> Sven
>>
>> ----- Ursprüngliche Mail -----
>> > Von: "Corey Nolet" <cj...@gmail.com>
>> > An: "user" <us...@accumulo.apache.org>
>> > Gesendet: Mittwoch, 16. September 2015 16:31:02
>> > Betreff: Re: Mini Accumulo Cluster reusing the directory
>>
>> > Sven,
>> >
>> > What version of Accumulo are you running? We have a ticket for this [1]
>> > which has had a lot of discussion on it. Christopher Tubbs mentioned
>> that
>> > he had gotten this to work.
>> >
>> > [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
>> >
>> > On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
>> sven.hodapp@scai.fraunhofer.de
>> >> wrote:
>> >
>> >> Hi there,
>> >>
>> >> is it possible for MiniAccumuloCluster to reuse a given directory?
>> >> Sadly, I haven't found anything in the docs?
>> >>
>> >> I’ll fire up my instance like this:
>> >>
>> >> val dict = new File("/tmp/accumulo-mini-cluster")
>> >> val accumulo = new MiniAccumuloCluster(dict, "test“)
>> >>
>> >> If I’ll restart my JVM it will raise a error like this:
>> >>
>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>> >> Directory /tmp/accumulo-mini-cluster is not empty
>> >>
>> >> It would be nice if the data can survive a JVM restart and the folder
>> >> structure must not be constructed every time.
>> >>
>> >> Thanks a lot!
>> >>
>> >> Regards,
>> >> Sven
>> >>
>> >> --
>> >> Sven Hodapp M.Sc.,
>> >> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> >> Department of Bioinformatics
>> >> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> >> sven.hodapp@scai.fraunhofer.de
>> >> www.scai.fraunhofer.de
>>
>
Re: Mini Accumulo Cluster reusing the directory
Posted by Christopher <ct...@apache.org>.
Hi Sven, I think Corey was mistaken. I don't recall working on that,
exactly.
On Wed, Sep 16, 2015, 10:40 Sven Hodapp <sv...@scai.fraunhofer.de>
wrote:
> Hi Corey,
>
> thanks for your reply and the link. Sounds good, if that will be available
> in the future!
> Is the code from Christopher somewhere deployed?
>
> Currently I'm using version 1.7
>
> Regards,
> Sven
>
> ----- Ursprüngliche Mail -----
> > Von: "Corey Nolet" <cj...@gmail.com>
> > An: "user" <us...@accumulo.apache.org>
> > Gesendet: Mittwoch, 16. September 2015 16:31:02
> > Betreff: Re: Mini Accumulo Cluster reusing the directory
>
> > Sven,
> >
> > What version of Accumulo are you running? We have a ticket for this [1]
> > which has had a lot of discussion on it. Christopher Tubbs mentioned that
> > he had gotten this to work.
> >
> > [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
> >
> > On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <
> sven.hodapp@scai.fraunhofer.de
> >> wrote:
> >
> >> Hi there,
> >>
> >> is it possible for MiniAccumuloCluster to reuse a given directory?
> >> Sadly, I haven't found anything in the docs?
> >>
> >> I’ll fire up my instance like this:
> >>
> >> val dict = new File("/tmp/accumulo-mini-cluster")
> >> val accumulo = new MiniAccumuloCluster(dict, "test“)
> >>
> >> If I’ll restart my JVM it will raise a error like this:
> >>
> >> Exception in thread "main" java.lang.IllegalArgumentException:
> >> Directory /tmp/accumulo-mini-cluster is not empty
> >>
> >> It would be nice if the data can survive a JVM restart and the folder
> >> structure must not be constructed every time.
> >>
> >> Thanks a lot!
> >>
> >> Regards,
> >> Sven
> >>
> >> --
> >> Sven Hodapp M.Sc.,
> >> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> >> Department of Bioinformatics
> >> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> >> sven.hodapp@scai.fraunhofer.de
> >> www.scai.fraunhofer.de
>
Re: Mini Accumulo Cluster reusing the directory
Posted by Sven Hodapp <sv...@scai.fraunhofer.de>.
Hi Corey,
thanks for your reply and the link. Sounds good, if that will be available in the future!
Is the code from Christopher somewhere deployed?
Currently I'm using version 1.7
Regards,
Sven
----- Ursprüngliche Mail -----
> Von: "Corey Nolet" <cj...@gmail.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Mittwoch, 16. September 2015 16:31:02
> Betreff: Re: Mini Accumulo Cluster reusing the directory
> Sven,
>
> What version of Accumulo are you running? We have a ticket for this [1]
> which has had a lot of discussion on it. Christopher Tubbs mentioned that
> he had gotten this to work.
>
> [1] https://issues.apache.org/jira/browse/ACCUMULO-1378
>
> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
>> wrote:
>
>> Hi there,
>>
>> is it possible for MiniAccumuloCluster to reuse a given directory?
>> Sadly, I haven't found anything in the docs?
>>
>> I’ll fire up my instance like this:
>>
>> val dict = new File("/tmp/accumulo-mini-cluster")
>> val accumulo = new MiniAccumuloCluster(dict, "test“)
>>
>> If I’ll restart my JVM it will raise a error like this:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> Directory /tmp/accumulo-mini-cluster is not empty
>>
>> It would be nice if the data can survive a JVM restart and the folder
>> structure must not be constructed every time.
>>
>> Thanks a lot!
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de
>> www.scai.fraunhofer.de
Re: Mini Accumulo Cluster reusing the directory
Posted by Corey Nolet <cj...@gmail.com>.
Sven,
What version of Accumulo are you running? We have a ticket for this [1]
which has had a lot of discussion on it. Christopher Tubbs mentioned that
he had gotten this to work.
[1] https://issues.apache.org/jira/browse/ACCUMULO-1378
On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
> wrote:
> Hi there,
>
> is it possible for MiniAccumuloCluster to reuse a given directory?
> Sadly, I haven't found anything in the docs?
>
> I’ll fire up my instance like this:
>
> val dict = new File("/tmp/accumulo-mini-cluster")
> val accumulo = new MiniAccumuloCluster(dict, "test“)
>
> If I’ll restart my JVM it will raise a error like this:
>
> Exception in thread "main" java.lang.IllegalArgumentException:
> Directory /tmp/accumulo-mini-cluster is not empty
>
> It would be nice if the data can survive a JVM restart and the folder
> structure must not be constructed every time.
>
> Thanks a lot!
>
> Regards,
> Sven
>
> --
> Sven Hodapp M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>