You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Stan Rosenberg <st...@gmail.com> on 2012/07/31 00:23:38 UTC
task jvm bootstrapping via distributed cache
Hi,
I am seeking a way to leverage hadoop's distributed cache in order to
ship jars that are required to bootstrap a task's jvm, i.e., before a
map/reduce task is launched.
As a concrete example, let's say that I need to launch with
'-javaagent:/path/profiler.jar'. In theory, the task tracker is
responsible for downloading cached files onto its local filesystem.
However, the absolute path to a given cached file is not known a
priori; however, we need the path in order to configure '-javaagent'.
Is this currently possible with the distributed cache? If not, is the
use case appealing enough to open a jira ticket?
Thanks,
stan
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
On Tue, Jul 31, 2012 at 7:26 PM, Michael Segel
<mi...@hotmail.com> wrote:
> Hi Stan,
>
> If I understood your question... you want to ship a jar to the nodes where the task will run prior to the start of the task?
>
> Not sure what it is you're trying to do...
> Your example isn't really clear.
Correct. I want to ship a jar to the task, but I need to know its
absolute path before the task jvm is launched.
As an example, -javaagent JVM option expects a jar path.
>
> See: http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/filecache/DistributedCache.html
>
> When you pull stuff out of the cache you get the path to the jar.
> Or you should be able to get it.
>
It would be too late at that point; the task tracker controls the
launching of the JVM. The path of the shipped jar need to be
available before the task is launched.
> Can you give a better example, there may be a different way to handle this...
>
Does the example above make sense?
Re: task jvm bootstrapping via distributed cache
Posted by Michael Segel <mi...@hotmail.com>.
Hi Stan,
If I understood your question... you want to ship a jar to the nodes where the task will run prior to the start of the task?
Not sure what it is you're trying to do...
Your example isn't really clear.
See: http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/filecache/DistributedCache.html
When you pull stuff out of the cache you get the path to the jar.
Or you should be able to get it.
I'm assuming you're doing this in your setup() method?
Can you give a better example, there may be a different way to handle this...
On Jul 31, 2012, at 3:50 PM, Stan Rosenberg <st...@gmail.com> wrote:
> Forwarding to common-user to hopefully get more exposure...
>
>
> ---------- Forwarded message ----------
> From: Stan Rosenberg <st...@gmail.com>
> Date: Tue, Jul 31, 2012 at 11:55 AM
> Subject: Re: task jvm bootstrapping via distributed cache
> To: mapreduce-user@hadoop.apache.org
>
>
> I am guessing this is either a well-known problem or an edge case. In
> any case, would it be a bad idea to designate predetermined output
> paths?
> E.g., DistributedCache.addCacheFileInto(uri, conf, outputPath) would
> attempt to copy the cached file into the specified path resolving to a
> task's local filesystem.
>
> Thanks,
>
> stan
>
> On Mon, Jul 30, 2012 at 6:23 PM, Stan Rosenberg
> <st...@gmail.com> wrote:
>> Hi,
>>
>> I am seeking a way to leverage hadoop's distributed cache in order to
>> ship jars that are required to bootstrap a task's jvm, i.e., before a
>> map/reduce task is launched.
>> As a concrete example, let's say that I need to launch with
>> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
>> responsible for downloading cached files onto its local filesystem.
>> However, the absolute path to a given cached file is not known a
>> priori; however, we need the path in order to configure '-javaagent'.
>>
>> Is this currently possible with the distributed cache? If not, is the
>> use case appealing enough to open a jira ticket?
>>
>> Thanks,
>>
>> stan
>
Fwd: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
Forwarding to common-user to hopefully get more exposure...
---------- Forwarded message ----------
From: Stan Rosenberg <st...@gmail.com>
Date: Tue, Jul 31, 2012 at 11:55 AM
Subject: Re: task jvm bootstrapping via distributed cache
To: mapreduce-user@hadoop.apache.org
I am guessing this is either a well-known problem or an edge case. In
any case, would it be a bad idea to designate predetermined output
paths?
E.g., DistributedCache.addCacheFileInto(uri, conf, outputPath) would
attempt to copy the cached file into the specified path resolving to a
task's local filesystem.
Thanks,
stan
On Mon, Jul 30, 2012 at 6:23 PM, Stan Rosenberg
<st...@gmail.com> wrote:
> Hi,
>
> I am seeking a way to leverage hadoop's distributed cache in order to
> ship jars that are required to bootstrap a task's jvm, i.e., before a
> map/reduce task is launched.
> As a concrete example, let's say that I need to launch with
> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> responsible for downloading cached files onto its local filesystem.
> However, the absolute path to a given cached file is not known a
> priori; however, we need the path in order to configure '-javaagent'.
>
> Is this currently possible with the distributed cache? If not, is the
> use case appealing enough to open a jira ticket?
>
> Thanks,
>
> stan
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
I am guessing this is either a well-known problem or an edge case. In
any case, would it be a bad idea to designate predetermined output
paths?
E.g., DistributedCache.addCacheFileInto(uri, conf, outputPath) would
attempt to copy the cached file into the specified path resolving to a
task's local filesystem.
Thanks,
stan
On Mon, Jul 30, 2012 at 6:23 PM, Stan Rosenberg
<st...@gmail.com> wrote:
> Hi,
>
> I am seeking a way to leverage hadoop's distributed cache in order to
> ship jars that are required to bootstrap a task's jvm, i.e., before a
> map/reduce task is launched.
> As a concrete example, let's say that I need to launch with
> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> responsible for downloading cached files onto its local filesystem.
> However, the absolute path to a given cached file is not known a
> priori; however, we need the path in order to configure '-javaagent'.
>
> Is this currently possible with the distributed cache? If not, is the
> use case appealing enough to open a jira ticket?
>
> Thanks,
>
> stan
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
On Fri, Aug 3, 2012 at 1:31 PM, Harsh J <ha...@cloudera.com> wrote:
>
> What this would do is merely take your passed -files jar (client-common) and
> symlink it into the JVM's working directory (the task's working directory)
> _before_ the JVM is begun, as "foo.jar". So if I pass additionally, JVM opts
> that refer to this foo.jar under ./, then it would work as you expect it to,
> as the JVM is begun from that directory (its CWD).
>
The fact that jvm is executed relative to task's cwd completely
escaped me :) Thanks for the clarification! It should definitely work.
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
Hi,
As I suspected, cache files are symlinked after a child JVM is
started: TaskRunner.setupWorkDir is being called from
org.apache.hadoop.mapred.Child.main.
This is unfortunate as it makes impossible to leverage distributed
cache for the purpose of deploying JVM agents. I could submit a jira
if there is any interest in getting this to work.
Otherwise, I'll think of some other hacks and use a distributed scp as
a last resort.
Thanks,
stan
On Thu, Jan 17, 2013 at 2:32 PM, Stan Rosenberg
<st...@gmail.com> wrote:
> Hi,
>
> I am back with my original problem. I am trying to bootstrap child
> JVM via -javaagent. I am doing what Harsh and Arun suggested, which
> also agrees with the documentation.
> In theory this should work, but it doesn't. Any ideas before I start
> digging into the code? Thanks.
>
> Here is the command I am using to test:
>
> hadoop jar /usr/lib/hadoop/hadoop-examples-0.20.2-cdh3u3.jar wordcount
> -files "core-tools-0.0.1-SNAPSHOT-common-assembly.jar#foo.jar"
> -Dmapred.map.child.java.opts="-javaagent:./foo.jar=classes=.*" test1
> output
>
> I can see the following (relevant) properties set in job.xml,
>
> mapred.cache.files=/user/srosenberg/.staging/job_201211061805_50132/files/core-tools-0.0.1-SNAPSHOT-common-assembly.jar#foo.jar
> mapred.create.symlink=yes
> mapred.map.child.java.opts=-javaagent:./foo.jar=classes=.*
>
> The map tasks fail with the following stdout/stderr output, resp.,
>
> Error occurred during initialization of VM
> agent library failed to init: instrument
>
> Error opening zip file or JAR manifest missing : ./foo.jar
>
> This seems like the jar is not symlinked into the current working
> directory of the child JVM; or perhaps the symlinking happens after
> the child JVM starts?
>
>
>
>
> On Fri, Aug 3, 2012 at 1:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> Stan,
>>
>> What Arun says would surely work.
>>
>> For instance, read this command:
>>
>> hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0.jar pi
>> -files
>> "share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.0.0.jar#foo.jar"
>> -Dmapred.child.java.opts="-javaagent:./foo.jar" 1 1
>>
>> What this would do is merely take your passed -files jar (client-common) and
>> symlink it into the JVM's working directory (the task's working directory)
>> _before_ the JVM is begun, as "foo.jar". So if I pass additionally, JVM opts
>> that refer to this foo.jar under ./, then it would work as you expect it to,
>> as the JVM is begun from that directory (its CWD).
>>
>> Do let us know if this solves it and also makes sense?
>>
>>
>> On Fri, Aug 3, 2012 at 10:02 PM, Stan Rosenberg <st...@gmail.com>
>> wrote:
>>>
>>> Arun,
>>>
>>> I don't believe the symlink is of help. The symlink is created in the
>>> task's current working directory (cwd), but I don't know what cwd is
>>> when I launch with 'hadoop jar ...'.
>>>
>>> Thanks,
>>>
>>> stan
>>>
>>> On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>>> > Stan,
>>> >
>>> > You can ask TT to create a symlink to your jar shipped via DistCache:
>>> >
>>> >
>>> > http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
>>> >
>>> > That should give you what you want.
>>> >
>>> > hth,
>>> > Arun
>>> >
>>> > On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am seeking a way to leverage hadoop's distributed cache in order to
>>> > ship jars that are required to bootstrap a task's jvm, i.e., before a
>>> > map/reduce task is launched.
>>> > As a concrete example, let's say that I need to launch with
>>> > '-javaagent:/path/profiler.jar'. In theory, the task tracker is
>>> > responsible for downloading cached files onto its local filesystem.
>>> > However, the absolute path to a given cached file is not known a
>>> > priori; however, we need the path in order to configure '-javaagent'.
>>> >
>>> > Is this currently possible with the distributed cache? If not, is the
>>> > use case appealing enough to open a jira ticket?
>>> >
>>> > Thanks,
>>> >
>>> > stan
>>> >
>>> >
>>> > --
>>> > Arun C. Murthy
>>> > Hortonworks Inc.
>>> > http://hortonworks.com/
>>> >
>>> >
>>
>>
>>
>>
>> --
>> Harsh J
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
Hi,
I am back with my original problem. I am trying to bootstrap child
JVM via -javaagent. I am doing what Harsh and Arun suggested, which
also agrees with the documentation.
In theory this should work, but it doesn't. Any ideas before I start
digging into the code? Thanks.
Here is the command I am using to test:
hadoop jar /usr/lib/hadoop/hadoop-examples-0.20.2-cdh3u3.jar wordcount
-files "core-tools-0.0.1-SNAPSHOT-common-assembly.jar#foo.jar"
-Dmapred.map.child.java.opts="-javaagent:./foo.jar=classes=.*" test1
output
I can see the following (relevant) properties set in job.xml,
mapred.cache.files=/user/srosenberg/.staging/job_201211061805_50132/files/core-tools-0.0.1-SNAPSHOT-common-assembly.jar#foo.jar
mapred.create.symlink=yes
mapred.map.child.java.opts=-javaagent:./foo.jar=classes=.*
The map tasks fail with the following stdout/stderr output, resp.,
Error occurred during initialization of VM
agent library failed to init: instrument
Error opening zip file or JAR manifest missing : ./foo.jar
This seems like the jar is not symlinked into the current working
directory of the child JVM; or perhaps the symlinking happens after
the child JVM starts?
On Fri, Aug 3, 2012 at 1:31 PM, Harsh J <ha...@cloudera.com> wrote:
> Stan,
>
> What Arun says would surely work.
>
> For instance, read this command:
>
> hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0.jar pi
> -files
> "share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.0.0.jar#foo.jar"
> -Dmapred.child.java.opts="-javaagent:./foo.jar" 1 1
>
> What this would do is merely take your passed -files jar (client-common) and
> symlink it into the JVM's working directory (the task's working directory)
> _before_ the JVM is begun, as "foo.jar". So if I pass additionally, JVM opts
> that refer to this foo.jar under ./, then it would work as you expect it to,
> as the JVM is begun from that directory (its CWD).
>
> Do let us know if this solves it and also makes sense?
>
>
> On Fri, Aug 3, 2012 at 10:02 PM, Stan Rosenberg <st...@gmail.com>
> wrote:
>>
>> Arun,
>>
>> I don't believe the symlink is of help. The symlink is created in the
>> task's current working directory (cwd), but I don't know what cwd is
>> when I launch with 'hadoop jar ...'.
>>
>> Thanks,
>>
>> stan
>>
>> On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> > Stan,
>> >
>> > You can ask TT to create a symlink to your jar shipped via DistCache:
>> >
>> >
>> > http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
>> >
>> > That should give you what you want.
>> >
>> > hth,
>> > Arun
>> >
>> > On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
>> >
>> > Hi,
>> >
>> > I am seeking a way to leverage hadoop's distributed cache in order to
>> > ship jars that are required to bootstrap a task's jvm, i.e., before a
>> > map/reduce task is launched.
>> > As a concrete example, let's say that I need to launch with
>> > '-javaagent:/path/profiler.jar'. In theory, the task tracker is
>> > responsible for downloading cached files onto its local filesystem.
>> > However, the absolute path to a given cached file is not known a
>> > priori; however, we need the path in order to configure '-javaagent'.
>> >
>> > Is this currently possible with the distributed cache? If not, is the
>> > use case appealing enough to open a jira ticket?
>> >
>> > Thanks,
>> >
>> > stan
>> >
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> >
>
>
>
>
> --
> Harsh J
Re: task jvm bootstrapping via distributed cache
Posted by Harsh J <ha...@cloudera.com>.
Stan,
What Arun says would surely work.
For instance, read this command:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0.jar pi
-files
"share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.0.0.jar#foo.jar"
-Dmapred.child.java.opts="-javaagent:./foo.jar" 1 1
What this would do is merely take your passed -files jar (client-common)
and symlink it into the JVM's working directory (the task's working
directory) _before_ the JVM is begun, as "foo.jar". So if I pass
additionally, JVM opts that refer to this foo.jar under ./, then it would
work as you expect it to, as the JVM is begun from that directory (its CWD).
Do let us know if this solves it and also makes sense?
On Fri, Aug 3, 2012 at 10:02 PM, Stan Rosenberg <st...@gmail.com>wrote:
> Arun,
>
> I don't believe the symlink is of help. The symlink is created in the
> task's current working directory (cwd), but I don't know what cwd is
> when I launch with 'hadoop jar ...'.
>
> Thanks,
>
> stan
>
> On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> > Stan,
> >
> > You can ask TT to create a symlink to your jar shipped via DistCache:
> >
> >
> http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
> >
> > That should give you what you want.
> >
> > hth,
> > Arun
> >
> > On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
> >
> > Hi,
> >
> > I am seeking a way to leverage hadoop's distributed cache in order to
> > ship jars that are required to bootstrap a task's jvm, i.e., before a
> > map/reduce task is launched.
> > As a concrete example, let's say that I need to launch with
> > '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> > responsible for downloading cached files onto its local filesystem.
> > However, the absolute path to a given cached file is not known a
> > priori; however, we need the path in order to configure '-javaagent'.
> >
> > Is this currently possible with the distributed cache? If not, is the
> > use case appealing enough to open a jira ticket?
> >
> > Thanks,
> >
> > stan
> >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
--
Harsh J
Re: task jvm bootstrapping via distributed cache
Posted by rahul p <ra...@gmail.com>.
Hi Arun,
I am new to hadoop n big data.
Can you help me start working on basics.my experience is into ETL and BI
DWH.
Rahul
On Aug 4, 2012 12:33 AM, "Stan Rosenberg" <st...@gmail.com> wrote:
> Arun,
>
> I don't believe the symlink is of help. The symlink is created in the
> task's current working directory (cwd), but I don't know what cwd is
> when I launch with 'hadoop jar ...'.
>
> Thanks,
>
> stan
>
> On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> > Stan,
> >
> > You can ask TT to create a symlink to your jar shipped via DistCache:
> >
> >
> http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
> >
> > That should give you what you want.
> >
> > hth,
> > Arun
> >
> > On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
> >
> > Hi,
> >
> > I am seeking a way to leverage hadoop's distributed cache in order to
> > ship jars that are required to bootstrap a task's jvm, i.e., before a
> > map/reduce task is launched.
> > As a concrete example, let's say that I need to launch with
> > '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> > responsible for downloading cached files onto its local filesystem.
> > However, the absolute path to a given cached file is not known a
> > priori; however, we need the path in order to configure '-javaagent'.
> >
> > Is this currently possible with the distributed cache? If not, is the
> > use case appealing enough to open a jira ticket?
> >
> > Thanks,
> >
> > stan
> >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
On Fri, Aug 3, 2012 at 4:19 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Just do -javaagent:./profiler.jar?
>
Yep, that should work. Thanks!
Re: task jvm bootstrapping via distributed cache
Posted by Arun C Murthy <ac...@hortonworks.com>.
Just do -javaagent:./profiler.jar?
On Aug 3, 2012, at 9:32 AM, Stan Rosenberg wrote:
> Arun,
>
> I don't believe the symlink is of help. The symlink is created in the
> task's current working directory (cwd), but I don't know what cwd is
> when I launch with 'hadoop jar ...'.
>
> Thanks,
>
> stan
>
> On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Stan,
>>
>> You can ask TT to create a symlink to your jar shipped via DistCache:
>>
>> http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
>>
>> That should give you what you want.
>>
>> hth,
>> Arun
>>
>> On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
>>
>> Hi,
>>
>> I am seeking a way to leverage hadoop's distributed cache in order to
>> ship jars that are required to bootstrap a task's jvm, i.e., before a
>> map/reduce task is launched.
>> As a concrete example, let's say that I need to launch with
>> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
>> responsible for downloading cached files onto its local filesystem.
>> However, the absolute path to a given cached file is not known a
>> priori; however, we need the path in order to configure '-javaagent'.
>>
>> Is this currently possible with the distributed cache? If not, is the
>> use case appealing enough to open a jira ticket?
>>
>> Thanks,
>>
>> stan
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: task jvm bootstrapping via distributed cache
Posted by Stan Rosenberg <st...@gmail.com>.
Arun,
I don't believe the symlink is of help. The symlink is created in the
task's current working directory (cwd), but I don't know what cwd is
when I launch with 'hadoop jar ...'.
Thanks,
stan
On Fri, Aug 3, 2012 at 2:39 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Stan,
>
> You can ask TT to create a symlink to your jar shipped via DistCache:
>
> http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
>
> That should give you what you want.
>
> hth,
> Arun
>
> On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
>
> Hi,
>
> I am seeking a way to leverage hadoop's distributed cache in order to
> ship jars that are required to bootstrap a task's jvm, i.e., before a
> map/reduce task is launched.
> As a concrete example, let's say that I need to launch with
> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> responsible for downloading cached files onto its local filesystem.
> However, the absolute path to a given cached file is not known a
> priori; however, we need the path in order to configure '-javaagent'.
>
> Is this currently possible with the distributed cache? If not, is the
> use case appealing enough to open a jira ticket?
>
> Thanks,
>
> stan
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: task jvm bootstrapping via distributed cache
Posted by Arun C Murthy <ac...@hortonworks.com>.
Stan,
You can ask TT to create a symlink to your jar shipped via DistCache:
http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache
That should give you what you want.
hth,
Arun
On Jul 30, 2012, at 3:23 PM, Stan Rosenberg wrote:
> Hi,
>
> I am seeking a way to leverage hadoop's distributed cache in order to
> ship jars that are required to bootstrap a task's jvm, i.e., before a
> map/reduce task is launched.
> As a concrete example, let's say that I need to launch with
> '-javaagent:/path/profiler.jar'. In theory, the task tracker is
> responsible for downloading cached files onto its local filesystem.
> However, the absolute path to a given cached file is not known a
> priori; however, we need the path in order to configure '-javaagent'.
>
> Is this currently possible with the distributed cache? If not, is the
> use case appealing enough to open a jira ticket?
>
> Thanks,
>
> stan
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/