You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Jason Trost <ja...@gmail.com> on 2012/05/02 17:31:52 UTC

Python client lib for Accumulo?

I noticed that there are no JIRAs for a python client
interface/lib/API for Accumulo.  How involved would it be to develop
AND maintain a python client for Accumulo?

I realize that Jython can be used, but I am interested in a native
python lib that can be use more broadly with systems that don't work
with Jython.

In order to do this, it seems like we would need to:
1. generate the python thrift bindings code (this is trivial)
2. develop and maintain the python glue code to use the thrift code
and python zookeeper code to interact with the various accumulo
components.  The current Java "glue" code looks quite long.  How often
does this code change (in terms of new features or changes in
protocol, not bug fixes)?

Ideally the python API would be very similar to the Java interface
(Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
Mutation, etc).

I guess what I am trying to get at is, does the Accumulo dev community
think it's worth the time and effort to develop and maintain a python
API?  I personally think it is in order to help with adoption and
integration with other systems (Django is the primary system I want to
be able to use with it).  I have some time to help this along, but I
don't think I have enough time to take this on alone.  Is anyone else
interested in working together on this?

Thanks,

--Jason

Re: Python client lib for Accumulo?

Posted by Jason Trost <ja...@gmail.com>.
I think a REST API helps and thats the approach we've taken
internally, but I think having a native python lib would be nicer
since we could write some of the data access and ingest code in python
instead of having to write a lot of that in Java and then use the REST
API.

On Wed, May 2, 2012 at 11:58 AM, David Medinets
<da...@gmail.com> wrote:
> There is a JIRA ticket for a REST interface to accumulo. Could that
> technique handle the integration that you're thinking of?
>
> On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com> wrote:
>> I noticed that there are no JIRAs for a python client
>> interface/lib/API for Accumulo.  How involved would it be to develop
>> AND maintain a python client for Accumulo?
>>
>> I realize that Jython can be used, but I am interested in a native
>> python lib that can be use more broadly with systems that don't work
>> with Jython.
>>
>> In order to do this, it seems like we would need to:
>> 1. generate the python thrift bindings code (this is trivial)
>> 2. develop and maintain the python glue code to use the thrift code
>> and python zookeeper code to interact with the various accumulo
>> components.  The current Java "glue" code looks quite long.  How often
>> does this code change (in terms of new features or changes in
>> protocol, not bug fixes)?
>>
>> Ideally the python API would be very similar to the Java interface
>> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
>> Mutation, etc).
>>
>> I guess what I am trying to get at is, does the Accumulo dev community
>> think it's worth the time and effort to develop and maintain a python
>> API?  I personally think it is in order to help with adoption and
>> integration with other systems (Django is the primary system I want to
>> be able to use with it).  I have some time to help this along, but I
>> don't think I have enough time to take this on alone.  Is anyone else
>> interested in working together on this?
>>
>> Thanks,
>>
>> --Jason

Re: Python client lib for Accumulo?

Posted by David Medinets <da...@gmail.com>.
There is a JIRA ticket for a REST interface to accumulo. Could that
technique handle the integration that you're thinking of?

On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com> wrote:
> I noticed that there are no JIRAs for a python client
> interface/lib/API for Accumulo.  How involved would it be to develop
> AND maintain a python client for Accumulo?
>
> I realize that Jython can be used, but I am interested in a native
> python lib that can be use more broadly with systems that don't work
> with Jython.
>
> In order to do this, it seems like we would need to:
> 1. generate the python thrift bindings code (this is trivial)
> 2. develop and maintain the python glue code to use the thrift code
> and python zookeeper code to interact with the various accumulo
> components.  The current Java "glue" code looks quite long.  How often
> does this code change (in terms of new features or changes in
> protocol, not bug fixes)?
>
> Ideally the python API would be very similar to the Java interface
> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
> Mutation, etc).
>
> I guess what I am trying to get at is, does the Accumulo dev community
> think it's worth the time and effort to develop and maintain a python
> API?  I personally think it is in order to help with adoption and
> integration with other systems (Django is the primary system I want to
> be able to use with it).  I have some time to help this along, but I
> don't think I have enough time to take this on alone.  Is anyone else
> interested in working together on this?
>
> Thanks,
>
> --Jason

Re: Python client lib for Accumulo?

Posted by Eric Newton <er...@gmail.com>.
The proxy would be co-located with the tablet servers, but as a standalone
service.

The old proxy code is has not been published, and is way too old to be
useful.  It would be easier and cleaner to start from scratch.

-Eric

On Wed, May 2, 2012 at 12:26 PM, Jason Trost <ja...@gmail.com> wrote:

> Eric,
>
> Where would the proxy service run?  I.e. how would it be deployed?  As
> a standalone server? Or bundled with each tablet server, etc?
>
> I think this approach might be the easiest from a maintenance and
> integration standpoint.
>
> Is your proxy code published anywhere?
>
> --Jason
>
> On Wed, May 2, 2012 at 11:58 AM, Eric Newton <er...@gmail.com>
> wrote:
> > Alternatively, you can create a Proxy service, which has a simple thrift
> > API, which would use the java client library.
> >
> > This is ACCUMULO-482.
> >
> > I wrote one once, but it didn't get much use, and died due to lack of
> > attention.
> >
> > -Eric
> >
> > On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com>
> wrote:
> >
> >> I noticed that there are no JIRAs for a python client
> >> interface/lib/API for Accumulo.  How involved would it be to develop
> >> AND maintain a python client for Accumulo?
> >>
> >> I realize that Jython can be used, but I am interested in a native
> >> python lib that can be use more broadly with systems that don't work
> >> with Jython.
> >>
> >> In order to do this, it seems like we would need to:
> >> 1. generate the python thrift bindings code (this is trivial)
> >> 2. develop and maintain the python glue code to use the thrift code
> >> and python zookeeper code to interact with the various accumulo
> >> components.  The current Java "glue" code looks quite long.  How often
> >> does this code change (in terms of new features or changes in
> >> protocol, not bug fixes)?
> >>
> >> Ideally the python API would be very similar to the Java interface
> >> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
> >> Mutation, etc).
> >>
> >> I guess what I am trying to get at is, does the Accumulo dev community
> >> think it's worth the time and effort to develop and maintain a python
> >> API?  I personally think it is in order to help with adoption and
> >> integration with other systems (Django is the primary system I want to
> >> be able to use with it).  I have some time to help this along, but I
> >> don't think I have enough time to take this on alone.  Is anyone else
> >> interested in working together on this?
> >>
> >> Thanks,
> >>
> >> --Jason
> >>
>

Re: Python client lib for Accumulo?

Posted by Jason Trost <ja...@gmail.com>.
Eric,

Where would the proxy service run?  I.e. how would it be deployed?  As
a standalone server? Or bundled with each tablet server, etc?

I think this approach might be the easiest from a maintenance and
integration standpoint.

Is your proxy code published anywhere?

--Jason

On Wed, May 2, 2012 at 11:58 AM, Eric Newton <er...@gmail.com> wrote:
> Alternatively, you can create a Proxy service, which has a simple thrift
> API, which would use the java client library.
>
> This is ACCUMULO-482.
>
> I wrote one once, but it didn't get much use, and died due to lack of
> attention.
>
> -Eric
>
> On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com> wrote:
>
>> I noticed that there are no JIRAs for a python client
>> interface/lib/API for Accumulo.  How involved would it be to develop
>> AND maintain a python client for Accumulo?
>>
>> I realize that Jython can be used, but I am interested in a native
>> python lib that can be use more broadly with systems that don't work
>> with Jython.
>>
>> In order to do this, it seems like we would need to:
>> 1. generate the python thrift bindings code (this is trivial)
>> 2. develop and maintain the python glue code to use the thrift code
>> and python zookeeper code to interact with the various accumulo
>> components.  The current Java "glue" code looks quite long.  How often
>> does this code change (in terms of new features or changes in
>> protocol, not bug fixes)?
>>
>> Ideally the python API would be very similar to the Java interface
>> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
>> Mutation, etc).
>>
>> I guess what I am trying to get at is, does the Accumulo dev community
>> think it's worth the time and effort to develop and maintain a python
>> API?  I personally think it is in order to help with adoption and
>> integration with other systems (Django is the primary system I want to
>> be able to use with it).  I have some time to help this along, but I
>> don't think I have enough time to take this on alone.  Is anyone else
>> interested in working together on this?
>>
>> Thanks,
>>
>> --Jason
>>

Re: Python client lib for Accumulo?

Posted by Eric Newton <er...@gmail.com>.
Alternatively, you can create a Proxy service, which has a simple thrift
API, which would use the java client library.

This is ACCUMULO-482.

I wrote one once, but it didn't get much use, and died due to lack of
attention.

-Eric

On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com> wrote:

> I noticed that there are no JIRAs for a python client
> interface/lib/API for Accumulo.  How involved would it be to develop
> AND maintain a python client for Accumulo?
>
> I realize that Jython can be used, but I am interested in a native
> python lib that can be use more broadly with systems that don't work
> with Jython.
>
> In order to do this, it seems like we would need to:
> 1. generate the python thrift bindings code (this is trivial)
> 2. develop and maintain the python glue code to use the thrift code
> and python zookeeper code to interact with the various accumulo
> components.  The current Java "glue" code looks quite long.  How often
> does this code change (in terms of new features or changes in
> protocol, not bug fixes)?
>
> Ideally the python API would be very similar to the Java interface
> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
> Mutation, etc).
>
> I guess what I am trying to get at is, does the Accumulo dev community
> think it's worth the time and effort to develop and maintain a python
> API?  I personally think it is in order to help with adoption and
> integration with other systems (Django is the primary system I want to
> be able to use with it).  I have some time to help this along, but I
> don't think I have enough time to take this on alone.  Is anyone else
> interested in working together on this?
>
> Thanks,
>
> --Jason
>

Re: Python client lib for Accumulo?

Posted by Keith Turner <ke...@deenlo.com>.
On Wed, May 2, 2012 at 11:31 AM, Jason Trost <ja...@gmail.com> wrote:
> I noticed that there are no JIRAs for a python client
> interface/lib/API for Accumulo.  How involved would it be to develop
> AND maintain a python client for Accumulo?
>
> I realize that Jython can be used, but I am interested in a native
> python lib that can be use more broadly with systems that don't work
> with Jython.
>
> In order to do this, it seems like we would need to:
> 1. generate the python thrift bindings code (this is trivial)
> 2. develop and maintain the python glue code to use the thrift code
> and python zookeeper code to interact with the various accumulo
> components.  The current Java "glue" code looks quite long.  How often
> does this code change (in terms of new features or changes in
> protocol, not bug fixes)?
>

I would advise against rewriting the accumulo client code in python.
The code that finds tablets, retries in case of failure, parallelizes
read/writes, etc is fairly complex.  I think the proxy option is best.
 David and Eric mentioned REST and Thrift proxies.

If we were to go to down the route of writing the client code in
another language, I think C++ with a C API would be the best option
because many language can easily bind to a C API.

> Ideally the python API would be very similar to the Java interface
> (Connector, Instance, Scanner, BatchScanner, BatchWriter, Key, Value,
> Mutation, etc).
>
> I guess what I am trying to get at is, does the Accumulo dev community
> think it's worth the time and effort to develop and maintain a python
> API?  I personally think it is in order to help with adoption and
> integration with other systems (Django is the primary system I want to
> be able to use with it).  I have some time to help this along, but I
> don't think I have enough time to take this on alone.  Is anyone else
> interested in working together on this?
>
> Thanks,
>
> --Jason