MapReduce - Libjars

[ka...@barclays.com]

Hi,

Within my map reduce programs; I am using an external JAVA library to help parse my raw files. When I am submitting my map reduce program, I am getting errors as the external class being referenced is not identified. Later, I explicitly specified the external jar being referenced with the "libjars" option. But I still continue to face problems as below. Any inputs to resolve the issue will be helpful.

12/06/06 13:36:37 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars

Regards,
Karan




This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial
Services Authority.

Re: MapReduce - Libjars

[Jagat Singh <ja...@gmail.com>]

Hello Karan,

Did you read this article

http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

You can have external jars in lib directory in your jar while packaging
your jar
or you can use DistributedCache<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html>
.

The syntax for doing with libjars is something like this

hadoop jar myjar.jar com.driver.ClassName -libjars
/home/thirdparty/grearJar.jar

It would be available at ${mapred.local.dir}/taskTracker/archive/${user.name
}/distcache/

Read that Cloudera article it would give you idea about number of ways of
doing it.

Besides this one previous discussion can also be helpful.
http://search-hadoop.com/m/gKLlP1hMjp21

Regards,

Jagat Singh



On Thu, Jun 7, 2012 at 10:36 AM, <ka...@barclays.com> wrote:

> Hi,
>
> Within my map reduce programs; I am using an external JAVA library to help
> parse my raw files. When I am submitting my map reduce program, I am
> getting errors as the external class being referenced is not identified.
> Later, I explicitly specified the external jar being referenced with the
> "libjars" option. But I still continue to face problems as below. Any
> inputs to resolve the issue will be helpful.
>
> 12/06/06 13:36:37 ERROR security.UserGroupInformation:
> PriviledgedActionException as:hadoop
> cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist: hdfs://ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
> Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException:
> Input path does not exist: hdfs://
> ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
>
> Regards,
> Karan
>
>
>
>
> This e-mail and any attachments are confidential and intended
> solely for the addressee and may also be privileged or exempt from
> disclosure under applicable law. If you are not the addressee, or
> have received this e-mail in error, please notify the sender
> immediately, delete it from your system and do not copy, disclose
> or otherwise act upon any part of this e-mail or its attachments.
>
> Internet communications are not guaranteed to be secure or
> virus-free.
> The Barclays Group does not accept responsibility for any loss
> arising from unauthorised access to, or interference with, any
> Internet communications by any third party, or from the
> transmission of any viruses. Replies to this e-mail may be
> monitored by the Barclays Group for operational or business
> reasons.
>
> Any opinion or other information in this e-mail or its attachments
> that does not relate to the business of the Barclays Group is
> personal to the sender and is not given or endorsed by the Barclays
> Group.
>
> Barclays Bank PLC. Registered in England and Wales (registered no.
> 1026167).
> Registered Office: 1 Churchill Place, London, E14 5HP, United
> Kingdom.
>
> Barclays Bank PLC is authorised and regulated by the Financial
> Services Authority.
>