You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bill de hÓra <bi...@dehora.net> on 2014/08/30 14:53:55 UTC
experience getting 0.8 perf scripts running
Hi,
On twitter recently [1] I said I'd send on a list of things that made
getting 0.8 setup for a quick bench awkward. Individually they're all
minor and I apologize if this seem like nitpicking. But in sum, getting
through them was a bit of a chore; it took maybe just short of a couple
of hours and a strong cup of coffee get to log stream nirvana :)
Anyway, here's a list of things that tripped me up -
- It seems like the 0.8 docs are in transition, or old docs have been
abandoned. There is some information on cwiki.apache.org and some on
kafka.apache.org. When I searched it was hit and miss which I landed on.
- The quickstart on cwiki.apache.org looks very wrong. It still shows
sbt, but the project builds with gradle, the kafka-create-topic.sh
params are wrong, the producer example's Partitioner has a generic param
but Partitioner in the code doesn't have one, etc.
- Some stuff on kafka.apache org also looked wrong, eg creating
topics in https://kafka.apache.org/08/quickstart.html, broken link to
http://kafka.apache.org/projects.html, the link back to the code example
on cwiki.
- The Kafka 0.8 binaries ship with perf scripts, but not perf lib.
- A download for the perf lib wasn't anywhere I could see on the site
(http://kafka.apache.org/downloads.html) - you'd have to know to search
maven, or checkout Kafka and build it, and then put the jar into the lib
area.
- The project doesn't work/build with java 8 afaict, or didn't for
me; there didn't seem to be a way to know that in advance.
- The project trunk failed to build due to some signing/license
checks (the diff I used to get things to build is here:
https://gist.github.com/dehora/7e3c0bd75bb2b5d87557).
- The javaapi coverage is partial. Initially I wasn't sure if I had
failed to import things from the right package before realizing I was
looking at bad docs.
- The messages flag on the perf script is required but not marked as
required.
In terms of what might help others, and I'd be happy to pitch in if
people thought they would make things easier, here's another list -
- Have working code examples for 0.8, and mention StringEncoder in
the docs for serializer.class.
- Delete wrong information. It seems like there should be just one
doc/wiki site and that cwiki.apache.org could be removed or marked dead
in big friendly letters.
- Ship the perf jar with the binary, or, remove the perf scripts and
create a separate perf distributable.
- Make the build work out of the box. It seems like the lic/singing
checks are for publication rather than local dev. (I haven't tried the
instructions to setup in an IDE yet).
Bill
[1] https://twitter.com/dehora/status/503207180544520193
Re: experience getting 0.8 perf scripts running
Posted by Jun Rao <ju...@gmail.com>.
Hi, Bill,
Thanks for sharing your experience. Yes, our documentation can definitely
be improved. As for your suggestion, I have
1. modified the producer example wiki (
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example)
to fix Partitioner.
2. marked the quickstart wiki page as obsolete.
3. filed KAFKA-1619 to fix the perf jar (the perf dir can actually be
removed).
As for signing during build, you can bypass it by specifying "-x
signArchives", which is described in README.md. Let me know if this needs
to be explicitly specified in any existing documentation.
Thanks,
Jun
On Sat, Aug 30, 2014 at 5:53 AM, Bill de hÓra <bi...@dehora.net> wrote:
> Hi,
>
> On twitter recently [1] I said I'd send on a list of things that made
> getting 0.8 setup for a quick bench awkward. Individually they're all minor
> and I apologize if this seem like nitpicking. But in sum, getting through
> them was a bit of a chore; it took maybe just short of a couple of hours
> and a strong cup of coffee get to log stream nirvana :)
>
> Anyway, here's a list of things that tripped me up -
>
> - It seems like the 0.8 docs are in transition, or old docs have been
> abandoned. There is some information on cwiki.apache.org and some on
> kafka.apache.org. When I searched it was hit and miss which I landed on.
>
> - The quickstart on cwiki.apache.org looks very wrong. It still shows
> sbt, but the project builds with gradle, the kafka-create-topic.sh params
> are wrong, the producer example's Partitioner has a generic param but
> Partitioner in the code doesn't have one, etc.
>
> - Some stuff on kafka.apache org also looked wrong, eg creating topics
> in https://kafka.apache.org/08/quickstart.html, broken link to
> http://kafka.apache.org/projects.html, the link back to the code example
> on cwiki.
>
> - The Kafka 0.8 binaries ship with perf scripts, but not perf lib.
>
> - A download for the perf lib wasn't anywhere I could see on the site (
> http://kafka.apache.org/downloads.html) - you'd have to know to search
> maven, or checkout Kafka and build it, and then put the jar into the lib
> area.
>
> - The project doesn't work/build with java 8 afaict, or didn't for me;
> there didn't seem to be a way to know that in advance.
>
> - The project trunk failed to build due to some signing/license checks
> (the diff I used to get things to build is here: https://gist.github.com/
> dehora/7e3c0bd75bb2b5d87557).
>
> - The javaapi coverage is partial. Initially I wasn't sure if I had
> failed to import things from the right package before realizing I was
> looking at bad docs.
>
> - The messages flag on the perf script is required but not marked as
> required.
>
>
> In terms of what might help others, and I'd be happy to pitch in if people
> thought they would make things easier, here's another list -
>
> - Have working code examples for 0.8, and mention StringEncoder in the
> docs for serializer.class.
>
> - Delete wrong information. It seems like there should be just one
> doc/wiki site and that cwiki.apache.org could be removed or marked dead
> in big friendly letters.
>
> - Ship the perf jar with the binary, or, remove the perf scripts and
> create a separate perf distributable.
>
> - Make the build work out of the box. It seems like the lic/singing
> checks are for publication rather than local dev. (I haven't tried the
> instructions to setup in an IDE yet).
>
> Bill
>
> [1] https://twitter.com/dehora/status/503207180544520193
>
Re: experience getting 0.8 perf scripts running
Posted by Joe Stein <jo...@stealth.ly>.
Yup, just created https://issues.apache.org/jira/browse/KAFKA-1622
On Mon, Sep 1, 2014 at 2:29 PM, Jun Rao <ju...@gmail.com> wrote:
> Hi, Joe,
>
> Since you added the signing part, is it possible to make it off by default
> (instead of having people do "-x signArchives")?
>
> Thanks,
>
> Jun
>
>
> On Mon, Sep 1, 2014 at 10:20 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hey Bill,
> >
> > Thanks for this very useful list.
> >
> > I think part of the problem is this. We have started versioning our
> > documentation with each release. We do this to ensure people on older
> > releases can still see the documentation. In the latest release we
> > migrated a bunch of stuff off the wiki to help it stay versioned with
> > the main docs--that was why you were seeing some content on the wiki
> > and some on the main site. So the older version of the docs point to
> > the wiki, and the newer docs are in html. Unfortunately Google isn't
> > very smart about this, and may well choose the top hit to be the doc
> > from the older release (similar thing happens with jdk javadocs).
> >
> > Here are some things I think we could do:
> > 1. Yes, in general we are keeping things in site docs rather than
> > wiki. The reason to retain the wiki version is so the docs on the
> > older version remain complete.
> > 2. Quickstart: you are using the 0.8.0 quickstart. It works with 0.8.0
> > but some details have changed for 0.8.1.1 (cli options, build
> > commands, etc).
> > 3. Ditto here. Good call on the projects page--looks like Jun fixed that.
> > 4. Oops, looks like Jun filed a JIRA for that.
> > 5. For Java 8, can you file a JIRA and include what you did and
> > whatever went wrong for you?
> > 6. Jun, can we default to not signing? That might be more intuitive...
> > 7. Can you elaborate on the javaapi coverage? I think you are saying
> > that the documentation is incomplete?
> > 8. Good point, that is a bug. Filed KAFKA-1621 for it.
> >
> > Each of the suggestions you made would be very useful, it would be
> > great if you would be willing to help us improve those things.
> >
> > -Jay
> >
> > On Sat, Aug 30, 2014 at 5:53 AM, Bill de hÓra <bi...@dehora.net> wrote:
> > > Hi,
> > >
> > > On twitter recently [1] I said I'd send on a list of things that made
> > > getting 0.8 setup for a quick bench awkward. Individually they're all
> > minor
> > > and I apologize if this seem like nitpicking. But in sum, getting
> through
> > > them was a bit of a chore; it took maybe just short of a couple of
> hours
> > and
> > > a strong cup of coffee get to log stream nirvana :)
> > >
> > > Anyway, here's a list of things that tripped me up -
> > >
> > > - It seems like the 0.8 docs are in transition, or old docs have been
> > > abandoned. There is some information on cwiki.apache.org and some on
> > > kafka.apache.org. When I searched it was hit and miss which I landed
> on.
> > >
> > > - The quickstart on cwiki.apache.org looks very wrong. It still
> shows
> > sbt,
> > > but the project builds with gradle, the kafka-create-topic.sh params
> are
> > > wrong, the producer example's Partitioner has a generic param but
> > > Partitioner in the code doesn't have one, etc.
> > >
> > > - Some stuff on kafka.apache org also looked wrong, eg creating
> topics
> > in
> > > https://kafka.apache.org/08/quickstart.html, broken link to
> > > http://kafka.apache.org/projects.html, the link back to the code
> > example on
> > > cwiki.
> > >
> > > - The Kafka 0.8 binaries ship with perf scripts, but not perf lib.
> > >
> > > - A download for the perf lib wasn't anywhere I could see on the site
> > > (http://kafka.apache.org/downloads.html) - you'd have to know to
> search
> > > maven, or checkout Kafka and build it, and then put the jar into the
> lib
> > > area.
> > >
> > > - The project doesn't work/build with java 8 afaict, or didn't for
> me;
> > > there didn't seem to be a way to know that in advance.
> > >
> > > - The project trunk failed to build due to some signing/license
> checks
> > > (the diff I used to get things to build is here:
> > > https://gist.github.com/dehora/7e3c0bd75bb2b5d87557).
> > >
> > > - The javaapi coverage is partial. Initially I wasn't sure if I had
> > failed
> > > to import things from the right package before realizing I was looking
> at
> > > bad docs.
> > >
> > > - The messages flag on the perf script is required but not marked as
> > > required.
> > >
> > >
> > > In terms of what might help others, and I'd be happy to pitch in if
> > people
> > > thought they would make things easier, here's another list -
> > >
> > > - Have working code examples for 0.8, and mention StringEncoder in
> the
> > > docs for serializer.class.
> > >
> > > - Delete wrong information. It seems like there should be just one
> > > doc/wiki site and that cwiki.apache.org could be removed or marked
> dead
> > in
> > > big friendly letters.
> > >
> > > - Ship the perf jar with the binary, or, remove the perf scripts and
> > > create a separate perf distributable.
> > >
> > > - Make the build work out of the box. It seems like the lic/singing
> > checks
> > > are for publication rather than local dev. (I haven't tried the
> > instructions
> > > to setup in an IDE yet).
> > >
> > > Bill
> > >
> > > [1] https://twitter.com/dehora/status/503207180544520193
> >
>
Re: experience getting 0.8 perf scripts running
Posted by Jun Rao <ju...@gmail.com>.
Hi, Joe,
Since you added the signing part, is it possible to make it off by default
(instead of having people do "-x signArchives")?
Thanks,
Jun
On Mon, Sep 1, 2014 at 10:20 AM, Jay Kreps <ja...@gmail.com> wrote:
> Hey Bill,
>
> Thanks for this very useful list.
>
> I think part of the problem is this. We have started versioning our
> documentation with each release. We do this to ensure people on older
> releases can still see the documentation. In the latest release we
> migrated a bunch of stuff off the wiki to help it stay versioned with
> the main docs--that was why you were seeing some content on the wiki
> and some on the main site. So the older version of the docs point to
> the wiki, and the newer docs are in html. Unfortunately Google isn't
> very smart about this, and may well choose the top hit to be the doc
> from the older release (similar thing happens with jdk javadocs).
>
> Here are some things I think we could do:
> 1. Yes, in general we are keeping things in site docs rather than
> wiki. The reason to retain the wiki version is so the docs on the
> older version remain complete.
> 2. Quickstart: you are using the 0.8.0 quickstart. It works with 0.8.0
> but some details have changed for 0.8.1.1 (cli options, build
> commands, etc).
> 3. Ditto here. Good call on the projects page--looks like Jun fixed that.
> 4. Oops, looks like Jun filed a JIRA for that.
> 5. For Java 8, can you file a JIRA and include what you did and
> whatever went wrong for you?
> 6. Jun, can we default to not signing? That might be more intuitive...
> 7. Can you elaborate on the javaapi coverage? I think you are saying
> that the documentation is incomplete?
> 8. Good point, that is a bug. Filed KAFKA-1621 for it.
>
> Each of the suggestions you made would be very useful, it would be
> great if you would be willing to help us improve those things.
>
> -Jay
>
> On Sat, Aug 30, 2014 at 5:53 AM, Bill de hÓra <bi...@dehora.net> wrote:
> > Hi,
> >
> > On twitter recently [1] I said I'd send on a list of things that made
> > getting 0.8 setup for a quick bench awkward. Individually they're all
> minor
> > and I apologize if this seem like nitpicking. But in sum, getting through
> > them was a bit of a chore; it took maybe just short of a couple of hours
> and
> > a strong cup of coffee get to log stream nirvana :)
> >
> > Anyway, here's a list of things that tripped me up -
> >
> > - It seems like the 0.8 docs are in transition, or old docs have been
> > abandoned. There is some information on cwiki.apache.org and some on
> > kafka.apache.org. When I searched it was hit and miss which I landed on.
> >
> > - The quickstart on cwiki.apache.org looks very wrong. It still shows
> sbt,
> > but the project builds with gradle, the kafka-create-topic.sh params are
> > wrong, the producer example's Partitioner has a generic param but
> > Partitioner in the code doesn't have one, etc.
> >
> > - Some stuff on kafka.apache org also looked wrong, eg creating topics
> in
> > https://kafka.apache.org/08/quickstart.html, broken link to
> > http://kafka.apache.org/projects.html, the link back to the code
> example on
> > cwiki.
> >
> > - The Kafka 0.8 binaries ship with perf scripts, but not perf lib.
> >
> > - A download for the perf lib wasn't anywhere I could see on the site
> > (http://kafka.apache.org/downloads.html) - you'd have to know to search
> > maven, or checkout Kafka and build it, and then put the jar into the lib
> > area.
> >
> > - The project doesn't work/build with java 8 afaict, or didn't for me;
> > there didn't seem to be a way to know that in advance.
> >
> > - The project trunk failed to build due to some signing/license checks
> > (the diff I used to get things to build is here:
> > https://gist.github.com/dehora/7e3c0bd75bb2b5d87557).
> >
> > - The javaapi coverage is partial. Initially I wasn't sure if I had
> failed
> > to import things from the right package before realizing I was looking at
> > bad docs.
> >
> > - The messages flag on the perf script is required but not marked as
> > required.
> >
> >
> > In terms of what might help others, and I'd be happy to pitch in if
> people
> > thought they would make things easier, here's another list -
> >
> > - Have working code examples for 0.8, and mention StringEncoder in the
> > docs for serializer.class.
> >
> > - Delete wrong information. It seems like there should be just one
> > doc/wiki site and that cwiki.apache.org could be removed or marked dead
> in
> > big friendly letters.
> >
> > - Ship the perf jar with the binary, or, remove the perf scripts and
> > create a separate perf distributable.
> >
> > - Make the build work out of the box. It seems like the lic/singing
> checks
> > are for publication rather than local dev. (I haven't tried the
> instructions
> > to setup in an IDE yet).
> >
> > Bill
> >
> > [1] https://twitter.com/dehora/status/503207180544520193
>
Re: experience getting 0.8 perf scripts running
Posted by Jay Kreps <ja...@gmail.com>.
Hey Bill,
Thanks for this very useful list.
I think part of the problem is this. We have started versioning our
documentation with each release. We do this to ensure people on older
releases can still see the documentation. In the latest release we
migrated a bunch of stuff off the wiki to help it stay versioned with
the main docs--that was why you were seeing some content on the wiki
and some on the main site. So the older version of the docs point to
the wiki, and the newer docs are in html. Unfortunately Google isn't
very smart about this, and may well choose the top hit to be the doc
from the older release (similar thing happens with jdk javadocs).
Here are some things I think we could do:
1. Yes, in general we are keeping things in site docs rather than
wiki. The reason to retain the wiki version is so the docs on the
older version remain complete.
2. Quickstart: you are using the 0.8.0 quickstart. It works with 0.8.0
but some details have changed for 0.8.1.1 (cli options, build
commands, etc).
3. Ditto here. Good call on the projects page--looks like Jun fixed that.
4. Oops, looks like Jun filed a JIRA for that.
5. For Java 8, can you file a JIRA and include what you did and
whatever went wrong for you?
6. Jun, can we default to not signing? That might be more intuitive...
7. Can you elaborate on the javaapi coverage? I think you are saying
that the documentation is incomplete?
8. Good point, that is a bug. Filed KAFKA-1621 for it.
Each of the suggestions you made would be very useful, it would be
great if you would be willing to help us improve those things.
-Jay
On Sat, Aug 30, 2014 at 5:53 AM, Bill de hÓra <bi...@dehora.net> wrote:
> Hi,
>
> On twitter recently [1] I said I'd send on a list of things that made
> getting 0.8 setup for a quick bench awkward. Individually they're all minor
> and I apologize if this seem like nitpicking. But in sum, getting through
> them was a bit of a chore; it took maybe just short of a couple of hours and
> a strong cup of coffee get to log stream nirvana :)
>
> Anyway, here's a list of things that tripped me up -
>
> - It seems like the 0.8 docs are in transition, or old docs have been
> abandoned. There is some information on cwiki.apache.org and some on
> kafka.apache.org. When I searched it was hit and miss which I landed on.
>
> - The quickstart on cwiki.apache.org looks very wrong. It still shows sbt,
> but the project builds with gradle, the kafka-create-topic.sh params are
> wrong, the producer example's Partitioner has a generic param but
> Partitioner in the code doesn't have one, etc.
>
> - Some stuff on kafka.apache org also looked wrong, eg creating topics in
> https://kafka.apache.org/08/quickstart.html, broken link to
> http://kafka.apache.org/projects.html, the link back to the code example on
> cwiki.
>
> - The Kafka 0.8 binaries ship with perf scripts, but not perf lib.
>
> - A download for the perf lib wasn't anywhere I could see on the site
> (http://kafka.apache.org/downloads.html) - you'd have to know to search
> maven, or checkout Kafka and build it, and then put the jar into the lib
> area.
>
> - The project doesn't work/build with java 8 afaict, or didn't for me;
> there didn't seem to be a way to know that in advance.
>
> - The project trunk failed to build due to some signing/license checks
> (the diff I used to get things to build is here:
> https://gist.github.com/dehora/7e3c0bd75bb2b5d87557).
>
> - The javaapi coverage is partial. Initially I wasn't sure if I had failed
> to import things from the right package before realizing I was looking at
> bad docs.
>
> - The messages flag on the perf script is required but not marked as
> required.
>
>
> In terms of what might help others, and I'd be happy to pitch in if people
> thought they would make things easier, here's another list -
>
> - Have working code examples for 0.8, and mention StringEncoder in the
> docs for serializer.class.
>
> - Delete wrong information. It seems like there should be just one
> doc/wiki site and that cwiki.apache.org could be removed or marked dead in
> big friendly letters.
>
> - Ship the perf jar with the binary, or, remove the perf scripts and
> create a separate perf distributable.
>
> - Make the build work out of the box. It seems like the lic/singing checks
> are for publication rather than local dev. (I haven't tried the instructions
> to setup in an IDE yet).
>
> Bill
>
> [1] https://twitter.com/dehora/status/503207180544520193