You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stas Oskin <st...@gmail.com> on 2010/06/17 13:02:40 UTC
KFS jar
Hi again.
I noticed a kfs jars in Hadoop distribution, does it mean the KFS is now
bult-in into Hadoop?
Regards.
Re: KFS jar
Posted by Steve Loughran <st...@apache.org>.
Stas Oskin wrote:
> Hi.
>
> I will point you at this presentation by my colleague Johannes Kirschnick,
>> Making Hadoop HA,
>>
>> http://www.slideshare.net/steve_l/high-availability-hadoop
>>
>> There's a performance graph on that slideset for small, virtualised
>> clusters, results in large physical clusters may vary
>>
>>
> Unfortunately they discuss HP IBRIX and not KFS.
I know, it just discusses why you'd want alternate filesystems.
Parascale have the patches for their FS to work with Hadoop in Jira too.
You'd need to find someone using KFS in their production environment,
and find out about things like server setup, disks/server, network
topology etc to see whether it would work for you, or you could do the
experiments yourself. The trouble is, experiments show things like
performance and robustness to managed outages (let's turn this server
off) and not the things you encounter in the field (a rack overheating,
switches misbehaving, race conditions surfacing). Everyone is on their
own there
>
>
>> just the snippet of my build which sets up the various classpath options
>> for different hadoop deployments. There's a lot of JARs there, but you can
>> strip the client-side down fairly well, even if Avro pushes out stuff you
>> don't need.
>>
>
> Now it's clear, thanks.
>
> Regards.
>
Re: KFS jar
Posted by Stas Oskin <st...@gmail.com>.
Hi.
I will point you at this presentation by my colleague Johannes Kirschnick,
> Making Hadoop HA,
>
> http://www.slideshare.net/steve_l/high-availability-hadoop
>
> There's a performance graph on that slideset for small, virtualised
> clusters, results in large physical clusters may vary
>
>
Unfortunately they discuss HP IBRIX and not KFS.
> just the snippet of my build which sets up the various classpath options
> for different hadoop deployments. There's a lot of JARs there, but you can
> strip the client-side down fairly well, even if Avro pushes out stuff you
> don't need.
>
Now it's clear, thanks.
Regards.
Re: KFS jar
Posted by Steve Loughran <st...@apache.org>.
Stas Oskin wrote:
> Hi.
>
> client side is, but if you don't want to talk to that FS, you don't need
>> that JAR.
>>
>
> How much KFS is faster then DFS?
no idea about speed or stability.
I will point you at this presentation by my colleague Johannes
Kirschnick, Making Hadoop HA,
http://www.slideshare.net/steve_l/high-availability-hadoop
There's a performance graph on that slideset for small, virtualised
clusters, results in large physical clusters may vary
>
> Also, does Hadoop supports the latest 0.5 version - or only the one the jar
> is named (i.e. 0.2)?
>
>
>> I'm attaching the bit of my ivy.xml I use, where I drop dependencies on
>> Jetty and slf4j and jasper because I pull jetty and jasper in from somewhere
>> else, and I provide a different logger for jetty, so removing the need for
>> slf4j
>>
>>
> I think it's the wrong address :).
just the snippet of my build which sets up the various classpath options
for different hadoop deployments. There's a lot of JARs there, but you
can strip the client-side down fairly well, even if Avro pushes out
stuff you don't need.
Re: KFS jar
Posted by Stas Oskin <st...@gmail.com>.
Hi.
client side is, but if you don't want to talk to that FS, you don't need
> that JAR.
>
How much KFS is faster then DFS?
Also, does Hadoop supports the latest 0.5 version - or only the one the jar
is named (i.e. 0.2)?
> I'm attaching the bit of my ivy.xml I use, where I drop dependencies on
> Jetty and slf4j and jasper because I pull jetty and jasper in from somewhere
> else, and I provide a different logger for jetty, so removing the need for
> slf4j
>
>
I think it's the wrong address :).
Regards.
Re: KFS jar
Posted by Steve Loughran <st...@apache.org>.
Stas Oskin wrote:
> Hi again.
>
> I noticed a kfs jars in Hadoop distribution, does it mean the KFS is now
> bult-in into Hadoop?
client side is, but if you don't want to talk to that FS, you don't need
that JAR.
I'm attaching the bit of my ivy.xml I use, where I drop dependencies on
Jetty and slf4j and jasper because I pull jetty and jasper in from
somewhere else, and I provide a different logger for jetty, so removing
the need for slf4j
<dependency org="org.apache.hadoop"
name="hadoop-core"
rev="${hadoop.version}"
changing="true"
conf="hadoop-core->default">
<exclude org="org.mortbay.jetty"/>
<exclude org="net.sf.kosmosfs"/>
<exclude org="com.thoughtworks.paranamer"/>
<exclude org="org.slf4j"/>
</dependency>
<dependency org="org.apache.hadoop"
name="hadoop-hdfs"
rev="${hadoop.version}"
changing="true"
conf="hadoop-core->default">
<exclude org="org.mortbay.jetty"/>
<exclude org="net.sf.kosmosfs"/>
<exclude org="com.thoughtworks.paranamer"/>
<exclude org="org.slf4j"/>
</dependency>
<dependency org="org.apache.hadoop"
name="hadoop-mapred"
rev="${hadoop.version}"
changing="true"
conf="hadoop-core->default">
<exclude org="org.mortbay.jetty"/>
<exclude org="net.sf.kosmosfs"/>
<exclude org="com.thoughtworks.paranamer"/>
<exclude org="org.slf4j"/>
</dependency>
<dependency org="org.apache.hadoop"
name="hadoop-mapred-examples"
rev="${hadoop.version}"
changing="true"
conf="hadoop-jars->default">
<exclude org="org.mortbay.jetty"/>
<exclude org="net.sf.kosmosfs"/>
<exclude org="com.thoughtworks.paranamer"/>
<exclude org="org.slf4j"/>
</dependency>
<dependency org="org.apache.hadoop"
name="hadoop-mapred-tools"
rev="${hadoop.version}"
changing="true"
conf="hadoop-jars->default">
<exclude org="org.mortbay.jetty"/>
<exclude org="net.sf.kosmosfs"/>
<exclude org="com.thoughtworks.paranamer"/>
<exclude org="org.slf4j"/>
</dependency>
<!-- client CLI. May also be used on the server to pass stuff to
forked processes-->
<dependency org="commons-cli"
name="commons-cli"
rev="${commons-cli.version}"
conf="redist->default"/>
<dependency org="xmlenc"
name="xmlenc"
rev="${xmlenc.version}"
conf="server->default"/>
<!--
Avro is the RPC mechanism,
-->
<dependency org="org.apache.hadoop"
name="avro"
rev="${avro.version}"
conf="redist->default">
<exclude org="com.thoughtworks.paranamer" />
<exclude org="org.slf4j"/>
<exclude org="org.mortbay.jetty"/>
</dependency>
<!-- used in Configuration now-->
<dependency org="org.codehaus.jackson"
name="jackson-mapper-asl"
rev="${jackson-asl.version}"
conf="redist->default"/>
<dependency org="org.codehaus.jackson"
name="jackson-core-asl"
rev="${jackson-asl.version}"
conf="redist->default"/>
<dependency org="com.thoughtworks.paranamer"
name="paranamer"
rev="${paranamer.version}"
conf="redist->default"/>
<!--Jetty 6 is pulled in from sf-jetty, along with JSP support
and implicitly, sf-ant
-->
<dependency org="org.smartfrog"
name="sf-jetty"
changing="true"
rev="latest.integration"
conf="server->default,jsp"/>
<!--Configuration: s3-client -->
<!--there are two jets3t projects in the repository; this one goes
up to 0.6 and
is assumed to be the live one-->
<dependency org="net.java.dev.jets3t"
name="jets3t"
rev="${jets3t.version}"
conf="s3-client->master"/>
<dependency org="net.java.dev.jets3t"
name="jets3t"
rev="${jets3t.version}"
conf="s3-client->master"/>
<!--to work with ftp:// fileystems, commons-net is needed -->
<dependency org="commons-net"
name="commons-net"
rev="${commons-net.version}"
conf="ftp->default"/>