You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Schubert Zhang <zs...@gmail.com> on 2009/10/19 19:02:34 UTC
hive-0.4.0 build
1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
2. Then I modified shims/ivy.xml and shims/build.xml to remove dependencies
of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download hadoop-0.20.0.
3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to add
0.20.1, but failed for md5, following is the errors.
[ivy:retrieve]
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] [FAILED ]
hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
(138662ms)
[ivy:retrieve] [FAILED ]
hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
(138662ms)
[ivy:retrieve] ==== hadoop-resolver: tried
[ivy:retrieve]
http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: FAILED DOWNLOADS ::
[ivy:retrieve] :: ^ see resolution messages for details ^ ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
I checked the
http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5,
and found the format of this file is different from other hadoop releases.
Please the releaser of hadoop and hive to have a check.
Schubert
Re: hive-0.4.0 build
Posted by Carl Steinbach <ca...@cloudera.com>.
The MD5 checksum file for the 0.20.1 tarball contains an MD5 checksum along
with
various SHA checksums and an RMD160 checksum:
hadoop-0.20.1.tar.gz: MD5 = 71 9E 16 9B 77 60 C1 68 44 1B 49 F4 05 85 5B
72
hadoop-0.20.1.tar.gz: SHA1 = 712A EE9C 279F 1031 1F83 657B 2B82 7ACA 0374
6613
hadoop-0.20.1.tar.gz: RMD160 = 4331 4350 27E9 E16D 055C F23F FFEF 1564 E206
B144
hadoop-0.20.1.tar.gz: SHA224 = A5E4CBE9 EBBE5FE1 2020F3F1 BFBC6D3C C77A8E9B
E6C6062C A6484BDB
...
The convention that most tools expect (including Ivy) is that a file named
xyz.tar.gz.md5 was generated by running the following command:
% md5sum xyz.tar.gz > xyz.tar.gz.md5
All of the other Hadoop releases on archive.apache.org follow this
convention, with
the exception of hadoop-0.20.1.
Can someone with access to the Apache archive repository please fix the
0.20.1
md5 checksum file?
Thanks.
Carl
On Mon, Oct 19, 2009 at 9:02 AM, Schubert Zhang <zs...@gmail.com> wrote:
> 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
> 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
> 2. Then I modified shims/ivy.xml and shims/build.xml to remove dependencies
> of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download hadoop-0.20.0.
>
> 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to add
> 0.20.1, but failed for md5, following is the errors.
>
> [ivy:retrieve]
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] ==== hadoop-resolver: tried
> [ivy:retrieve]
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: FAILED DOWNLOADS ::
> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> I checked the
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5
> ,
> and found the format of this file is different from other hadoop releases.
>
> Please the releaser of hadoop and hive to have a check.
>
> Schubert
>
Re: hive-0.4.0 build
Posted by Matt Pestritto <ma...@pestritto.com>.
FYI. Same thing happened to me.
My temp workaround was:
cd ~/.ant/cache/hadoop/core/sources
wget
http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
Then rebuild hive. Ivy will not try to download the source.
Thanks
-Matt
On Mon, Oct 19, 2009 at 6:41 PM, Zheng Shao <zs...@gmail.com> wrote:
> Hi Schubert,
>
> I am not an expert on ivy, but is it possible to change the md5 check logic
> in ivy?
> We might want to support both .md5 file format. If you are interested in
> going this route, please open a JIRA and we can work together on that.
>
> Also, for the time being, you might want to just skip the md5 check if
> possible.
>
> Zheng
>
>
> On Mon, Oct 19, 2009 at 10:02 AM, Schubert Zhang <zs...@gmail.com>wrote:
>
>> 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
>> 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
>> 2. Then I modified shims/ivy.xml and shims/build.xml to remove
>> dependencies of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download
>> hadoop-0.20.0.
>>
>> 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to
>> add 0.20.1, but failed for md5, following is the errors.
>>
>> [ivy:retrieve]
>> [ivy:retrieve] :: problems summary ::
>> [ivy:retrieve] :::: WARNINGS
>> [ivy:retrieve] [FAILED ]
>> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
>> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
>> (138662ms)
>> [ivy:retrieve] [FAILED ]
>> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
>> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
>> (138662ms)
>> [ivy:retrieve] ==== hadoop-resolver: tried
>> [ivy:retrieve]
>> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
>> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
>> [ivy:retrieve] :: FAILED DOWNLOADS ::
>> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
>> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
>> [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
>> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
>> [ivy:retrieve]
>> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>>
>> I checked the
>> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5,
>> and found the format of this file is different from other hadoop releases.
>>
>> Please the releaser of hadoop and hive to have a check.
>>
>> Schubert
>>
>
>
>
> --
> Yours,
> Zheng
>
Re: hive-0.4.0 build
Posted by Zheng Shao <zs...@gmail.com>.
Hi Schubert,
I am not an expert on ivy, but is it possible to change the md5 check logic
in ivy?
We might want to support both .md5 file format. If you are interested in
going this route, please open a JIRA and we can work together on that.
Also, for the time being, you might want to just skip the md5 check if
possible.
Zheng
On Mon, Oct 19, 2009 at 10:02 AM, Schubert Zhang <zs...@gmail.com> wrote:
> 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
> 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
> 2. Then I modified shims/ivy.xml and shims/build.xml to remove dependencies
> of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download hadoop-0.20.0.
>
> 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to add
> 0.20.1, but failed for md5, following is the errors.
>
> [ivy:retrieve]
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] ==== hadoop-resolver: tried
> [ivy:retrieve]
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: FAILED DOWNLOADS ::
> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> I checked the
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5,
> and found the format of this file is different from other hadoop releases.
>
> Please the releaser of hadoop and hive to have a check.
>
> Schubert
>
--
Yours,
Zheng
Re: hive-0.4.0 build
Posted by Schubert Zhang <zs...@gmail.com>.
Todd,
Yes, it works when build against 0.20.0. I just mention this issue for your
check.
Thanks.
Schubert
On Tue, Oct 20, 2009 at 1:12 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Schubert,
>
> Regarding the Hive problem - you should be able to build against 0.20.0 and
> then run against 0.20.1. Within a release (0.20.*) the APIs are completely
> compatible, so you can compile any external application against one and run
> against another.
>
> Regarding the format of the md5 file, I'll defer to Owen who I believe cut
> the release.
>
> -Todd
>
> On Mon, Oct 19, 2009 at 10:02 AM, Schubert Zhang <zs...@gmail.com>
> wrote:
>
> > 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
> > 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
> > 2. Then I modified shims/ivy.xml and shims/build.xml to remove
> dependencies
> > of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download
> hadoop-0.20.0.
> >
> > 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to
> add
> > 0.20.1, but failed for md5, following is the errors.
> >
> > [ivy:retrieve]
> > [ivy:retrieve] :: problems summary ::
> > [ivy:retrieve] :::: WARNINGS
> > [ivy:retrieve] [FAILED ]
> > hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> > expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> > (138662ms)
> > [ivy:retrieve] [FAILED ]
> > hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> > expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> > (138662ms)
> > [ivy:retrieve] ==== hadoop-resolver: tried
> > [ivy:retrieve]
> >
> >
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
> > [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:retrieve] :: FAILED DOWNLOADS ::
> > [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> > [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
> > [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:retrieve]
> > [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> >
> > I checked the
> >
> >
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5
> > ,
> > and found the format of this file is different from other hadoop
> releases.
> >
> > Please the releaser of hadoop and hive to have a check.
> >
> > Schubert
> >
>
Re: hive-0.4.0 build
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Schubert,
Regarding the Hive problem - you should be able to build against 0.20.0 and
then run against 0.20.1. Within a release (0.20.*) the APIs are completely
compatible, so you can compile any external application against one and run
against another.
Regarding the format of the md5 file, I'll defer to Owen who I believe cut
the release.
-Todd
On Mon, Oct 19, 2009 at 10:02 AM, Schubert Zhang <zs...@gmail.com> wrote:
> 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
> 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
> 2. Then I modified shims/ivy.xml and shims/build.xml to remove dependencies
> of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download hadoop-0.20.0.
>
> 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to add
> 0.20.1, but failed for md5, following is the errors.
>
> [ivy:retrieve]
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] ==== hadoop-resolver: tried
> [ivy:retrieve]
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: FAILED DOWNLOADS ::
> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> I checked the
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5
> ,
> and found the format of this file is different from other hadoop releases.
>
> Please the releaser of hadoop and hive to have a check.
>
> Schubert
>
Re: hive-0.4.0 build
Posted by Carl Steinbach <ca...@cloudera.com>.
The MD5 checksum file for the 0.20.1 tarball contains an MD5 checksum along
with
various SHA checksums and an RMD160 checksum:
hadoop-0.20.1.tar.gz: MD5 = 71 9E 16 9B 77 60 C1 68 44 1B 49 F4 05 85 5B
72
hadoop-0.20.1.tar.gz: SHA1 = 712A EE9C 279F 1031 1F83 657B 2B82 7ACA 0374
6613
hadoop-0.20.1.tar.gz: RMD160 = 4331 4350 27E9 E16D 055C F23F FFEF 1564 E206
B144
hadoop-0.20.1.tar.gz: SHA224 = A5E4CBE9 EBBE5FE1 2020F3F1 BFBC6D3C C77A8E9B
E6C6062C A6484BDB
...
The convention that most tools expect (including Ivy) is that a file named
xyz.tar.gz.md5 was generated by running the following command:
% md5sum xyz.tar.gz > xyz.tar.gz.md5
All of the other Hadoop releases on archive.apache.org follow this
convention, with
the exception of hadoop-0.20.1.
Can someone with access to the Apache archive repository please fix the
0.20.1
md5 checksum file?
Thanks.
Carl
On Mon, Oct 19, 2009 at 9:02 AM, Schubert Zhang <zs...@gmail.com> wrote:
> 1. When I build hive-0.4.0, ivy would try to download hadoop 0.17.2.1,
> 0.18.3, 0.19.0 and 0.20.0. But always fail for 0.17.2.1.
> 2. Then I modified shims/ivy.xml and shims/build.xml to remove dependencies
> of 0.17.2.1, 0.18.3, 0.19.0. It works find to only download hadoop-0.20.0.
>
> 3. Now, I want do depend hadoop-0.20.1, I modified the above two xml to add
> 0.20.1, but failed for md5, following is the errors.
>
> [ivy:retrieve]
> [ivy:retrieve] :: problems summary ::
> [ivy:retrieve] :::: WARNINGS
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] [FAILED ]
> hadoop#core;0.20.1!hadoop.tar.gz(source): invalid md5:
> expected=hadoop-0.20.1.tar.gz: computed=719e169b7760c168441b49f405855b72
> (138662ms)
> [ivy:retrieve] ==== hadoop-resolver: tried
> [ivy:retrieve]
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: FAILED DOWNLOADS ::
> [ivy:retrieve] :: ^ see resolution messages for details ^ ::
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
> [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:retrieve]
> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> I checked the
>
> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5
> ,
> and found the format of this file is different from other hadoop releases.
>
> Please the releaser of hadoop and hive to have a check.
>
> Schubert
>