You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Thomas Koch <th...@koch.ro> on 2010/02/24 09:53:02 UTC
additional source only release tarball
Hi,
I've packaged hadoop for Debian. (It's still in the upload queue for
verification[1]).
One common annoyance when packaging java applications for a Free Software
distribution is the necessity to repackage the upstream tarball. The
repackaging is necessary, because Debian may only distribute binary files
build from source that's also available from Debian.
So we build the jar/war files ourselfes to make sure there's nothing we don't
have the sources for.
It would take one (annoying and time consuming) step less for packagers, if
java upstream projects would release an additional tarball without any binary
files or third party code.
I'm asking you first, because many other projects (like zookeeper) took or
take hadoop as an example for their build infrastructure.
For your orientation, these are the patterns that I used to filter the hadoop
tarball: (Usable with tar --exclude)
"*.jar",
"uming.*",
"prototype.js",
"config.sub",
"config.guess",
"ltmain.sh",
"Makefile.in",
"configure",
"aclocal.m4",
"config.h.in",
"install-sh",
"autom4te.cache",
"depcomp",
"missing",
"pipes/compile",
"src/contrib/eclipse-plugin/resources/*.jpg",
"src/contrib/eclipse-plugin/resources/*.png",
"src/contrib/eclipse-plugin/resources/*.gif",
"hadoop-0.20.1/src/core/org/apache/hadoop/record/compiler/generated/*.java",
"hadoop-0.20.1/src/docs/cn/build",
"hadoop-0.20.1/c++",
"hadoop-0.20.1/contrib",
"hadoop-0.20.1/lib/native",
"hadoop-0.20.1/librecordio",
"hadoop-0.20.1/src/contrib/thriftfs/gen-*",
"hadoop-0.20.1/docs",
There were different reasons why stuff needed to be filtered:
- unclear license (uming.*)
- unclear origin (images in the eclipse plugin)
- precompiled documentation / code / hadoop binaries
- pregenerated C/C++ automake files
- third party libraries (prototype.js, lib/*.jar)
If you'd be willing to release an additional tarball like this, I'd provide
the necessary patch to your build.xml
[1] http://ftp-master.debian.org/new.html
Best regards,
Thomas Koch, http://www.koch.ro
Automation/Performance Testing on Hadoop
Posted by Devender <dk...@qasource.com>.
Hi,
I had few questions on Hadoop,
1. Is there any way to extract data from Hadoop (data will be in XML)
and building customer usage models?
2. Creating tests using those models to drive HTTP, SMTP, etc.
3. Automating the running of those tests on a variety of test matrices
at varying load levels to determine system maxima while monitoring
system resource parameters to identify bottlenecks.
4. producing graphs/charts based on Excel, perl, web GUI, etc.
5. Which performance tools can be used for this?
6. Which Automation tools can be used for this?
Kindly provide your feedback if any one have any information?
Regards,
Dev
Re: additional source only release tarball
Posted by Patrick Hunt <ph...@apache.org>.
Ok, that's good to know. Thanks.
Patrick
Doug Cutting wrote:
> Patrick Hunt wrote:
>> Ah, thanks for clarify that Doug. To take it a bit further, when you
>> say "bug" you really mean "serious breach of Apache process/rules",
>> would that be valid? i.e. it would be something that the responsible
>> Apache team should work to address with highest of priority.
>
> To some degree that depends on the Apache project. I don't know of a
> project that does not create release tags and that would accept an
> incorrect one lightly. That said, release tags are not required nor
> authoritative: the thing that counts is the signed artifact.
>
> I'd certainly encourage developers to leverage tags when convenient
> e.g., for automated testing against and comparison with prior releases,
> for IDE source browsing, etc. But if someone wants to package an
> alternate distribution of an Apache release, I think they're better
> starting from the release artifact than the tag. The artifact can be
> validated against the signature at http://www.apache.org/dist/, while
> there's currently no good means of validating the contents of a tag. I
> suppose one could rebuild the tarball from the tag and try to validate
> its checksum against that at http://www.apache.org/dist/, but that seems
> both fragile and less secure.
>
> Doug
Re: additional source only release tarball
Posted by Doug Cutting <cu...@apache.org>.
Patrick Hunt wrote:
> Ah, thanks for clarify that Doug. To take it a bit further, when you say
> "bug" you really mean "serious breach of Apache process/rules", would
> that be valid? i.e. it would be something that the responsible Apache
> team should work to address with highest of priority.
To some degree that depends on the Apache project. I don't know of a
project that does not create release tags and that would accept an
incorrect one lightly. That said, release tags are not required nor
authoritative: the thing that counts is the signed artifact.
I'd certainly encourage developers to leverage tags when convenient
e.g., for automated testing against and comparison with prior releases,
for IDE source browsing, etc. But if someone wants to package an
alternate distribution of an Apache release, I think they're better
starting from the release artifact than the tag. The artifact can be
validated against the signature at http://www.apache.org/dist/, while
there's currently no good means of validating the contents of a tag. I
suppose one could rebuild the tarball from the tag and try to validate
its checksum against that at http://www.apache.org/dist/, but that seems
both fragile and less secure.
Doug
Re: additional source only release tarball
Posted by Patrick Hunt <ph...@apache.org>.
Doug Cutting wrote:
> Patrick Hunt wrote:
>> If you use the release tag from SVN you are guaranteed to get the
>> same source as is contained in the release artifact.
>
> "Guarantee" is a strong word. You certainly should get the same source,
> but the source that's voted on, that's checksummed and that's signed is
> the release artifact, not the SVN tag. It's a bug if this is not the
> same as the release tag, but not impossible.
Ah, thanks for clarify that Doug. To take it a bit further, when you say
"bug" you really mean "serious breach of Apache process/rules", would
that be valid? i.e. it would be something that the responsible Apache
team should work to address with highest of priority.
Patrick
ps. I do like this Avro idea of a separate src artifact (re the signing
esp).
Re: additional source only release tarball
Posted by Doug Cutting <cu...@apache.org>.
Patrick Hunt wrote:
> Thomas, not sure if this solves all your issues but have you considered
> using SVN rather than the release tarball?
FWIW, in Avro we've moved away from a single release tarball that
includes dependencies. Rather the primary release artifact is a source
tarball generated with 'svn export'. In addition we distribute derived
artifacts as conveniences, like a documentation tarball, a java jar &
pom for upload to maven, a java executable jar for command-line tools, a
python egg, a ruby gem, C & C++ pristine tarballs, etc.
> If you use the release tag from SVN you are guaranteed to get the
> same source as is contained in the release artifact.
"Guarantee" is a strong word. You certainly should get the same source,
but the source that's voted on, that's checksummed and that's signed is
the release artifact, not the SVN tag. It's a bug if this is not the
same as the release tag, but not impossible.
Doug
Re: additional source only release tarball
Posted by Patrick Hunt <ph...@apache.org>.
Thomas, not sure if this solves all your issues but have you considered
using SVN rather than the release tarball? The latest versions of Hadoop
and ZooKeeper have moved to ant/ivy, while HBase has moved to Maven (and
ZooKeeper has been considering moving to Maven at some point). As a
result I'm not sure how much overlap there will be going forward wrt a
common build...
If you use the release tag from SVN you are guaranteed to get the same
source as is contained in the release artifact. However the release
artifact contains additional things like lib dependencies (typically the
results of "ant package" or "ant tar")
By moving to SVN you would guarantee that you have the same source but
since ivy/maven are used to pull the dependencies (for the most part,
there may be some exceptions) you'd essentially get what you are looking
for.
Patrick
Steve Loughran wrote:
> Thomas Koch wrote:
>> Hi Steve,
>>
>>> One thing I've discussed before is the whole notion of having some
>>> hadoop-redist project whose aim is to produce useful
>>> packaging/installation of the artifacts of the main Hadoop projects.
>> you may be interested in project-builder.org. It's a tool to integrate
>> the building of distribution packages in a continuous integration
>> workflow. There has been a presentation at the European Free Software
>> conference FOSDEM:
>>
>> Continuous Packaging with Project-Builder.org
>> http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/
>>
>> Best regards,
>>
>> Thomas Koch, http://www.koch.ro
>
> thanks, I'll look at it. I do have the CI tools creating RPMs right now,
> scp-ing to VMs, though the fact that the rpm tools (especially rpm -qf
> to verify that a file exists/is managed) doesn't return error/success
> codes, so I have to grab the ssh output and scan for error strings,
> which we all think is wrong.
Re: additional source only release tarball
Posted by Steve Loughran <st...@apache.org>.
Thomas Koch wrote:
> Hi Steve,
>
>> One thing I've discussed before is the whole notion of having some
>> hadoop-redist project whose aim is to produce useful
>> packaging/installation of the artifacts of the main Hadoop projects.
> you may be interested in project-builder.org. It's a tool to integrate the
> building of distribution packages in a continuous integration workflow. There
> has been a presentation at the European Free Software conference FOSDEM:
>
> Continuous Packaging with Project-Builder.org
> http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
thanks, I'll look at it. I do have the CI tools creating RPMs right now,
scp-ing to VMs, though the fact that the rpm tools (especially rpm -qf
to verify that a file exists/is managed) doesn't return error/success
codes, so I have to grab the ssh output and scan for error strings,
which we all think is wrong.
Re: additional source only release tarball
Posted by Thomas Koch <th...@koch.ro>.
Hi Steve,
> One thing I've discussed before is the whole notion of having some
> hadoop-redist project whose aim is to produce useful
> packaging/installation of the artifacts of the main Hadoop projects.
you may be interested in project-builder.org. It's a tool to integrate the
building of distribution packages in a continuous integration workflow. There
has been a presentation at the European Free Software conference FOSDEM:
Continuous Packaging with Project-Builder.org
http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/
Best regards,
Thomas Koch, http://www.koch.ro
Re: additional source only release tarball
Posted by Steve Loughran <st...@apache.org>.
Arun C Murthy wrote:
> This seems reasonable, anyone else?
>
> Thomas, could you, please, open a jira and attach your patch to
> build.xml there? I have some comments on some of your exclude patterns,
> but we can have that discussion on the jira.
>
> thanks,
> Arun
One thing I've discussed before is the whole notion of having some
hadoop-redist project whose aim is to produce useful
packaging/installation of the artifacts of the main Hadoop projects.
-RPMs
-Debs
-stuff to help the various CM tools work w/ Hadoop
-things to help is deploy and test at scale
This may seem trivial, but it isn't. Right now my test process is
* create my own RPMs
* scp them to a machine with some scripts (that I don't run) to
* ssh to that machine, run some scripts and create a VM disk image
containing the RPMs and the installer scripts
* bring up the controller front end to the infrastructure
Then I run a junit test
* Have htmlunit start talking to a web ui
* through it, ask for 20 VMs
* go get my 16th coffee of the day, assuming this is test run #14
(first two coffees are at home, one before the school run, the other
while doing the first round of emails)
* have the controller wait for the VMs to come up
* as they do, for each one, run a thread that SSHs in and actually
pushes out the Hadoop config based on the hostnames given to the
different VMs
* run terasort
* grab the logs
* teardown
* merge the logs in with the rest of the junit output
* try and work out why the test failed
oh, and everything from the 20 vms onwards is started by junit test
To make life more fun, that bit of [1] has to spin waiting for the DNS
names to resolve, the machines to boot enough for SSH to work, then SSH
in and spin there waiting for the installed programs to be on the path
and then the CM tooling to be live. And feed the log of this not just to
the local log4j log but (somehow) stream it back over the (remote) web
ui that has to then live update the web page (this is surprisingly hard)
Here's a bit of it, excluding the coffees
PortUtils.waitForHostnameToResolve(hostname,
factory.getPortConnectTimeout(),
SLEEP_TIME_FOR_HOSTNAME_RESOLUTION);
PortUtils.checkPort(hostname, factory.getPort(),
factory.getPortConnectTimeout());
//some stuff creating SSH sessions, SCP-ing over files skipped
//make a few attempts to find the startup command
for (int i = 0; i < STARTUP_LOCATE_ATTEMPTS; i++) {
commandsList.add("which " + SF_START + " || sleep " +
STARTUP_SLEEP_TIME);
}
commandsList.add("which " + SF_START + " || echo " +
ERROR_NO_EXECUTABLE + SF_START);
String sfPing = makeSFCommand(SF_PING) + " " + "localhost";
for (int i = 0; i < STARTUP_PING_ATTEMPTS; i++) {
commandsList.add(sfPing + " || sleep " +
STARTUP_PING_SLEEP_TIME);
}
commandsList.add(sshCommand);
commandsList.add("sleep " + SLEEP_TIME);
if (!factory.isKeepFiles()) {
commandsList.add("rm " + desttempfile);
}
commandsList.add("exit");
sshExec(session, commandsList);
See that? fun. JUnit test cases that fail in interesting ways. Really
interesting ways. Sometimes it's the infrastructure, sometimes it's my
code. Sometimes, because I am working with SVN_HEAD, it's Hadoop.
Problem is, it takes a while to track down, especially if the VMs are no
longer there at the end of the run.
Right now one of the problems is that none of Hadoop's JSP pages work. I
can submit jobs, talk to the FS, etc, but all the JSP pages are failing
and the root of every web is /webapps ; I think it's something that
Todd's also seeing, but I can' t make it go away, not without fixing how
Hadoop locates/registers webapps.
Returning to the idea of a deployment project, having more tests that
can be run against production clusters would be good: all the tests
should be redistributables that you can deploy/run. There is something
needed to QA a cluster, and ideally identify the approximate area where
problems lie (e.g. DNS playing up, rDNS not working, clocks out of sync,
etc)
thoughts?
-steve
Re: additional source only release tarball
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
This seems reasonable, anyone else?
Thomas, could you, please, open a jira and attach your patch to
build.xml there? I have some comments on some of your exclude
patterns, but we can have that discussion on the jira.
thanks,
Arun
On Feb 24, 2010, at 12:53 AM, Thomas Koch wrote:
> Hi,
>
> I've packaged hadoop for Debian. (It's still in the upload queue for
> verification[1]).
> One common annoyance when packaging java applications for a Free
> Software
> distribution is the necessity to repackage the upstream tarball. The
> repackaging is necessary, because Debian may only distribute binary
> files
> build from source that's also available from Debian.
> So we build the jar/war files ourselfes to make sure there's nothing
> we don't
> have the sources for.
> It would take one (annoying and time consuming) step less for
> packagers, if
> java upstream projects would release an additional tarball without
> any binary
> files or third party code.
> I'm asking you first, because many other projects (like zookeeper)
> took or
> take hadoop as an example for their build infrastructure.
> For your orientation, these are the patterns that I used to filter
> the hadoop
> tarball: (Usable with tar --exclude)
>
> "*.jar",
> "uming.*",
> "prototype.js",
> "config.sub",
> "config.guess",
> "ltmain.sh",
> "Makefile.in",
> "configure",
> "aclocal.m4",
> "config.h.in",
> "install-sh",
> "autom4te.cache",
> "depcomp",
> "missing",
> "pipes/compile",
> "src/contrib/eclipse-plugin/resources/*.jpg",
> "src/contrib/eclipse-plugin/resources/*.png",
> "src/contrib/eclipse-plugin/resources/*.gif",
> "hadoop-0.20.1/src/core/org/apache/hadoop/record/compiler/generated/
> *.java",
> "hadoop-0.20.1/src/docs/cn/build",
> "hadoop-0.20.1/c++",
> "hadoop-0.20.1/contrib",
> "hadoop-0.20.1/lib/native",
> "hadoop-0.20.1/librecordio",
> "hadoop-0.20.1/src/contrib/thriftfs/gen-*",
> "hadoop-0.20.1/docs",
>
> There were different reasons why stuff needed to be filtered:
> - unclear license (uming.*)
> - unclear origin (images in the eclipse plugin)
> - precompiled documentation / code / hadoop binaries
> - pregenerated C/C++ automake files
> - third party libraries (prototype.js, lib/*.jar)
>
> If you'd be willing to release an additional tarball like this, I'd
> provide
> the necessary patch to your build.xml
>
> [1] http://ftp-master.debian.org/new.html
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro