You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Keith Bourgoin <ke...@gmail.com> on 2015/06/08 21:24:06 UTC
Anouncing PyKafka!
Hi Kafka folks,
I'm happy to announce the 1.0 release of PyKafka
<https://github.com/parsely/pykafka>, a new Python driver for Kafka!
This is an upgrade of an older library, named samsa, which was only
compatible with Kafka 0.7.x. We've spent the last few months upgrading it
to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
more efficient.
The biggest difference between PyKafka and other Python drivers is the
inclusion of a balanced consumer implementation. We implemented the same
balancing algorithm as the Java driver and PyKafka handles consumer
registration via Zookeeper like the Java driver.
We at Parse.ly have been using this driver in production for a few weeks
and are confident in its stability. Even so, we're looking to add more
features over the next couple months and are always trying to find ways to
improve performance.
You can find the project on PyPI <https://pypi.python.org/pypi/pykafka/>,
Github <https://github.com/parsely/pykafka> and our mailing list
<https://groups.google.com/forum/#%21forum/pykafka-user>. We also wrote a blog
post
<http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/>
with a bit more info.
Cheers,
Keith.
Re: Anouncing PyKafka!
Posted by Dillian Murphey <cr...@gmail.com>.
EXCELLENT WORK. Thank you!
On Wed, Jun 10, 2015 at 11:19 AM, Keith Bourgoin <ke...@gmail.com>
wrote:
> I would guess within a week or two, but it's hard to say for sure. Emmett
> Butler, the other main contributor, has been hard at work on it when he has
> the time.
>
> On Wed, Jun 10, 2015 at 12:18 PM Vijay Patil <vi...@gmail.com>
> wrote:
>
> > Thanks a lot Keith for detailed explanation, that will surely help me.
> > Another question, I can see AsyncProducer is not yet implemented, any
> idea
> > by when it can be available?
> >
> > Thanks,
> > Vijay
> >
> > On 10 June 2015 at 18:42, Keith Bourgoin <ke...@gmail.com>
> wrote:
> >
> > > Hi Vijay,
> > >
> > > PyKafka support custom partitioners so that you can control how the
> > > producer distributes messages. Looking at it now, I realize the
> > > documentation around this is a bit lacking, so I'll add that. In the
> > > meantime, here's an example of how to use the hashing partitioner which
> > is
> > > included with PyKafka.
> > >
> > > client = pykafka.KafkaClient()
> > >
> > > # Producer with random_partitioner
> > > prod = client.topics['foo'].get_producer()
> > > prod.produce(['foo', 'bar', 'baz'])
> > >
> > > # Producer with hashing_partitioner
> > > prod =
> > >
> >
> client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
> > > prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
> > >
> > >
> > >
> > > Cheers,
> > >
> > > Keith
> > >
> > >
> > > On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vi...@gmail.com>
> > > wrote:
> > >
> > > > Thanks,
> > > > a question, does it support keyed Producer?
> > > > I mean how it works when I have multiple partitions? Will it be able
> to
> > > > identify partition based of the key which I pass?
> > > >
> > > > On 9 June 2015 at 00:54, Keith Bourgoin <ke...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Kafka folks,
> > > > >
> > > > > I'm happy to announce the 1.0 release of PyKafka
> > > > > <https://github.com/parsely/pykafka>, a new Python driver for
> Kafka!
> > > > >
> > > > > This is an upgrade of an older library, named samsa, which was only
> > > > > compatible with Kafka 0.7.x. We've spent the last few months
> > upgrading
> > > it
> > > > > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner
> > and
> > > > > more efficient.
> > > > >
> > > > > The biggest difference between PyKafka and other Python drivers is
> > the
> > > > > inclusion of a balanced consumer implementation. We implemented the
> > > same
> > > > > balancing algorithm as the Java driver and PyKafka handles consumer
> > > > > registration via Zookeeper like the Java driver.
> > > > >
> > > > > We at Parse.ly have been using this driver in production for a few
> > > weeks
> > > > > and are confident in its stability. Even so, we're looking to add
> > more
> > > > > features over the next couple months and are always trying to find
> > ways
> > > > to
> > > > > improve performance.
> > > > >
> > > > > You can find the project on PyPI <
> > > https://pypi.python.org/pypi/pykafka/
> > > > >,
> > > > > Github <https://github.com/parsely/pykafka> and our mailing list
> > > > > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also
> > > wrote
> > > > a
> > > > > blog
> > > > > post
> > > > > <
> > > > >
> > > >
> > >
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > > > > >
> > > > > with a bit more info.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Keith.
> > > > >
> > > >
> > >
> >
>
Re: Anouncing PyKafka!
Posted by Keith Bourgoin <ke...@gmail.com>.
I would guess within a week or two, but it's hard to say for sure. Emmett
Butler, the other main contributor, has been hard at work on it when he has
the time.
On Wed, Jun 10, 2015 at 12:18 PM Vijay Patil <vi...@gmail.com>
wrote:
> Thanks a lot Keith for detailed explanation, that will surely help me.
> Another question, I can see AsyncProducer is not yet implemented, any idea
> by when it can be available?
>
> Thanks,
> Vijay
>
> On 10 June 2015 at 18:42, Keith Bourgoin <ke...@gmail.com> wrote:
>
> > Hi Vijay,
> >
> > PyKafka support custom partitioners so that you can control how the
> > producer distributes messages. Looking at it now, I realize the
> > documentation around this is a bit lacking, so I'll add that. In the
> > meantime, here's an example of how to use the hashing partitioner which
> is
> > included with PyKafka.
> >
> > client = pykafka.KafkaClient()
> >
> > # Producer with random_partitioner
> > prod = client.topics['foo'].get_producer()
> > prod.produce(['foo', 'bar', 'baz'])
> >
> > # Producer with hashing_partitioner
> > prod =
> >
> client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
> > prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
> >
> >
> >
> > Cheers,
> >
> > Keith
> >
> >
> > On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vi...@gmail.com>
> > wrote:
> >
> > > Thanks,
> > > a question, does it support keyed Producer?
> > > I mean how it works when I have multiple partitions? Will it be able to
> > > identify partition based of the key which I pass?
> > >
> > > On 9 June 2015 at 00:54, Keith Bourgoin <ke...@gmail.com>
> > wrote:
> > >
> > > > Hi Kafka folks,
> > > >
> > > > I'm happy to announce the 1.0 release of PyKafka
> > > > <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
> > > >
> > > > This is an upgrade of an older library, named samsa, which was only
> > > > compatible with Kafka 0.7.x. We've spent the last few months
> upgrading
> > it
> > > > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner
> and
> > > > more efficient.
> > > >
> > > > The biggest difference between PyKafka and other Python drivers is
> the
> > > > inclusion of a balanced consumer implementation. We implemented the
> > same
> > > > balancing algorithm as the Java driver and PyKafka handles consumer
> > > > registration via Zookeeper like the Java driver.
> > > >
> > > > We at Parse.ly have been using this driver in production for a few
> > weeks
> > > > and are confident in its stability. Even so, we're looking to add
> more
> > > > features over the next couple months and are always trying to find
> ways
> > > to
> > > > improve performance.
> > > >
> > > > You can find the project on PyPI <
> > https://pypi.python.org/pypi/pykafka/
> > > >,
> > > > Github <https://github.com/parsely/pykafka> and our mailing list
> > > > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also
> > wrote
> > > a
> > > > blog
> > > > post
> > > > <
> > > >
> > >
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > > > >
> > > > with a bit more info.
> > > >
> > > > Cheers,
> > > >
> > > > Keith.
> > > >
> > >
> >
>
Re: Anouncing PyKafka!
Posted by Vijay Patil <vi...@gmail.com>.
Thanks a lot Keith for detailed explanation, that will surely help me.
Another question, I can see AsyncProducer is not yet implemented, any idea
by when it can be available?
Thanks,
Vijay
On 10 June 2015 at 18:42, Keith Bourgoin <ke...@gmail.com> wrote:
> Hi Vijay,
>
> PyKafka support custom partitioners so that you can control how the
> producer distributes messages. Looking at it now, I realize the
> documentation around this is a bit lacking, so I'll add that. In the
> meantime, here's an example of how to use the hashing partitioner which is
> included with PyKafka.
>
> client = pykafka.KafkaClient()
>
> # Producer with random_partitioner
> prod = client.topics['foo'].get_producer()
> prod.produce(['foo', 'bar', 'baz'])
>
> # Producer with hashing_partitioner
> prod =
> client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
> prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
>
>
>
> Cheers,
>
> Keith
>
>
> On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vi...@gmail.com>
> wrote:
>
> > Thanks,
> > a question, does it support keyed Producer?
> > I mean how it works when I have multiple partitions? Will it be able to
> > identify partition based of the key which I pass?
> >
> > On 9 June 2015 at 00:54, Keith Bourgoin <ke...@gmail.com>
> wrote:
> >
> > > Hi Kafka folks,
> > >
> > > I'm happy to announce the 1.0 release of PyKafka
> > > <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
> > >
> > > This is an upgrade of an older library, named samsa, which was only
> > > compatible with Kafka 0.7.x. We've spent the last few months upgrading
> it
> > > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
> > > more efficient.
> > >
> > > The biggest difference between PyKafka and other Python drivers is the
> > > inclusion of a balanced consumer implementation. We implemented the
> same
> > > balancing algorithm as the Java driver and PyKafka handles consumer
> > > registration via Zookeeper like the Java driver.
> > >
> > > We at Parse.ly have been using this driver in production for a few
> weeks
> > > and are confident in its stability. Even so, we're looking to add more
> > > features over the next couple months and are always trying to find ways
> > to
> > > improve performance.
> > >
> > > You can find the project on PyPI <
> https://pypi.python.org/pypi/pykafka/
> > >,
> > > Github <https://github.com/parsely/pykafka> and our mailing list
> > > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also
> wrote
> > a
> > > blog
> > > post
> > > <
> > >
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > > >
> > > with a bit more info.
> > >
> > > Cheers,
> > >
> > > Keith.
> > >
> >
>
Re: Anouncing PyKafka!
Posted by Keith Bourgoin <ke...@gmail.com>.
Hi Vijay,
PyKafka support custom partitioners so that you can control how the
producer distributes messages. Looking at it now, I realize the
documentation around this is a bit lacking, so I'll add that. In the
meantime, here's an example of how to use the hashing partitioner which is
included with PyKafka.
client = pykafka.KafkaClient()
# Producer with random_partitioner
prod = client.topics['foo'].get_producer()
prod.produce(['foo', 'bar', 'baz'])
# Producer with hashing_partitioner
prod = client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
Cheers,
Keith
On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vi...@gmail.com>
wrote:
> Thanks,
> a question, does it support keyed Producer?
> I mean how it works when I have multiple partitions? Will it be able to
> identify partition based of the key which I pass?
>
> On 9 June 2015 at 00:54, Keith Bourgoin <ke...@gmail.com> wrote:
>
> > Hi Kafka folks,
> >
> > I'm happy to announce the 1.0 release of PyKafka
> > <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
> >
> > This is an upgrade of an older library, named samsa, which was only
> > compatible with Kafka 0.7.x. We've spent the last few months upgrading it
> > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
> > more efficient.
> >
> > The biggest difference between PyKafka and other Python drivers is the
> > inclusion of a balanced consumer implementation. We implemented the same
> > balancing algorithm as the Java driver and PyKafka handles consumer
> > registration via Zookeeper like the Java driver.
> >
> > We at Parse.ly have been using this driver in production for a few weeks
> > and are confident in its stability. Even so, we're looking to add more
> > features over the next couple months and are always trying to find ways
> to
> > improve performance.
> >
> > You can find the project on PyPI <https://pypi.python.org/pypi/pykafka/
> >,
> > Github <https://github.com/parsely/pykafka> and our mailing list
> > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also wrote
> a
> > blog
> > post
> > <
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > >
> > with a bit more info.
> >
> > Cheers,
> >
> > Keith.
> >
>
Re: Anouncing PyKafka!
Posted by Vijay Patil <vi...@gmail.com>.
Thanks,
a question, does it support keyed Producer?
I mean how it works when I have multiple partitions? Will it be able to
identify partition based of the key which I pass?
On 9 June 2015 at 00:54, Keith Bourgoin <ke...@gmail.com> wrote:
> Hi Kafka folks,
>
> I'm happy to announce the 1.0 release of PyKafka
> <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
>
> This is an upgrade of an older library, named samsa, which was only
> compatible with Kafka 0.7.x. We've spent the last few months upgrading it
> to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
> more efficient.
>
> The biggest difference between PyKafka and other Python drivers is the
> inclusion of a balanced consumer implementation. We implemented the same
> balancing algorithm as the Java driver and PyKafka handles consumer
> registration via Zookeeper like the Java driver.
>
> We at Parse.ly have been using this driver in production for a few weeks
> and are confident in its stability. Even so, we're looking to add more
> features over the next couple months and are always trying to find ways to
> improve performance.
>
> You can find the project on PyPI <https://pypi.python.org/pypi/pykafka/>,
> Github <https://github.com/parsely/pykafka> and our mailing list
> <https://groups.google.com/forum/#%21forum/pykafka-user>. We also wrote a
> blog
> post
> <
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> >
> with a bit more info.
>
> Cheers,
>
> Keith.
>
Re: Anouncing PyKafka!
Posted by Magnus Edenhill <ma...@edenhill.se>.
That's exciting news, looks very promising!
I've added PyKafka to the Clients wiki page.
Good work!
2015-06-08 21:24 GMT+02:00 Keith Bourgoin <ke...@gmail.com>:
> Hi Kafka folks,
>
> I'm happy to announce the 1.0 release of PyKafka
> <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
>
> This is an upgrade of an older library, named samsa, which was only
> compatible with Kafka 0.7.x. We've spent the last few months upgrading it
> to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
> more efficient.
>
> The biggest difference between PyKafka and other Python drivers is the
> inclusion of a balanced consumer implementation. We implemented the same
> balancing algorithm as the Java driver and PyKafka handles consumer
> registration via Zookeeper like the Java driver.
>
> We at Parse.ly have been using this driver in production for a few weeks
> and are confident in its stability. Even so, we're looking to add more
> features over the next couple months and are always trying to find ways to
> improve performance.
>
> You can find the project on PyPI <https://pypi.python.org/pypi/pykafka/>,
> Github <https://github.com/parsely/pykafka> and our mailing list
> <https://groups.google.com/forum/#%21forum/pykafka-user>. We also wrote a
> blog
> post
> <
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> >
> with a bit more info.
>
> Cheers,
>
> Keith.
>