You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Dylan Hutchison <dh...@cs.washington.edu> on 2016/08/02 02:45:13 UTC
Tang in Yarn - overriding a NamedParameter
Hi Reefies,
I have a Reef application that runs on Yarn. Reef is responsible for
distributing some jars to the various containers that start in the Yarn
cluster.
From what I can tell, Reef uses a JobUploader to put the jars in a location
visible to all the containers. I need to change this location from the
default.
The default location is controlled by
@NamedParameter(doc = "The job submission directory prefix.",
default_value = "/tmp")
public final class JobSubmissionDirectoryPrefix implements Name<String> {
}
I would like to change this to a custom directory, because /tmp is not a
globally visible location in my cluster.
How do I do so?
Thanks,
Dylan Hutchison
Tang in Yarn - overriding a NamedParameter
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
---------- Forwarded message ----------
From: "Dylan Hutchison" <dh...@cs.washington.edu>
Date: Aug 1, 2016 7:45 PM
Subject: Tang in Yarn - overriding a NamedParameter
To: <de...@reef.apache.org>
Cc:
Hi Reefies,
I have a Reef application that runs on Yarn. Reef is responsible for
distributing some jars to the various containers that start in the Yarn
cluster.
From what I can tell, Reef uses a JobUploader to put the jars in a location
visible to all the containers. I need to change this location from the
default.
The default location is controlled by
@NamedParameter(doc = "The job submission directory prefix.",
default_value = "/tmp")
public final class JobSubmissionDirectoryPrefix implements Name<String> {
}
I would like to change this to a custom directory, because /tmp is not a
globally visible location in my cluster.
How do I do so?
Thanks,
Dylan Hutchison
Re: Tang in Yarn - overriding a NamedParameter
Posted by John Yang <jo...@gmail.com>.
I see. Glad you found a solution. :)
John
On Sat, Aug 6, 2016 at 7:07 AM, Dylan Hutchison <dh...@cs.washington.edu>
wrote:
> Hi John,
>
> The exception occurs after the evaluator spawns. I think you have a point:
> Reef may be working properly, but there may be other problems with the
> application.
>
> For now I got around the issue by installing HDFS and switching the file
> system to that. The directory "/tmp" is globally visible on HDFS, so
> everything worked there. It's a shame I couldn't make it work on the
> Lustre file system (in which /tmp is *not* mounted), but I think this is an
> application problem instead of a Reef problem.
>
> Cheers, Dylan
>
> On Wed, Aug 3, 2016 at 11:28 PM, John Yang <jo...@gmail.com> wrote:
>
> > Hi Dylan,
> >
> > Does the exception occur before the Evaluator process is spawned or
> after?
> > If it's before, then I think YARN is failing to locate the directory.
> > If it's after, I really don't have a good idea as the parameter seem to
> be
> > for Driver only, being in the package "org.apache.reef.driver.
> parameters".
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Aug 3, 2016 at 4:55 AM, Dylan Hutchison <
> > dhutchis@cs.washington.edu>
> > wrote:
> >
> > > Thanks Gyewon and John. I made some progress: the driver now sets the
> > > DriverJobSubmissionDirectory
> > > <
> > > https://reef.apache.org/apidocs/0.15.0/org/apache/
> > reef/javabridge/generic/Launch.DriverJobSubmissionDirectory.html
> > > >
> > > to the appropriate value and the Application Master recognizes it.
> > >
> > > Unfortunately, this information does not propagate to the containers
> that
> > > the Driver requests, when it requests an Evaluator. The containers
> > > continue to use the default value "/tmp". The message the driver
> > receives
> > > from the container for an allocated evaluator is a
> FileNotFoundException,
> > > because the evaluator tries to use the wrong directory and can't find
> the
> > > jars that Reef is trying to send to the evaluators.
> > >
> > > Is there a way to propogate the DriverJobSubmissionDirectory globally
> to
> > > all Yarn processes? I've looked at the EvaluatorRequest
> > > <
> > > https://reef.apache.org/apidocs/0.15.0/org/apache/
> reef/driver/evaluator/
> > EvaluatorRequest.html
> > > >
> > > API but can't find anything.
> > >
> > > Thanks, Dylan
> > >
> > >
> > > On Mon, Aug 1, 2016 at 11:40 PM, Gyewon Lee <st...@gmail.com>
> > wrote:
> > >
> > > > Hi, Dylan.
> > > >
> > > > In Tang NamedParameter, "default_value" is ignored when you
> explicitly
> > > set
> > > > your own value via JavaConfigurationBuilder (using methods like
> > > > bindNamedParameter()) or ConfigurationModule (using method 'set()' in
> > > > John's answer). So, you don't need to modify REEF source code for
> > > changing
> > > > default_value. In your case, John's solution will work quite nicely
> > > because
> > > > there is a predefined ConfigurationModule
> > (YarnDriverConfiguration.CONF).
> > > >
> > > > In other cases where the ConfigurationModule is not set for your
> case,
> > > then
> > > > you can bind the value for NamedParameters, like
> > > >
> > > > JavaConfigurationBuilder cb =
> > > > Tang.Factory.getTang().newConfigurationBuilder();
> > > > cb.bindNamedParameter(JobSubmissionDirectoryPrefix.class,
> > > > "/path/to/job/submission/directory/prefix");
> > > > existingConfiguration.merge(cb.build());
> > > >
> > > > You don't need to do like that in this case, but I hope this would
> help
> > > in
> > > > your future REEF programming.
> > > >
> > > > Best regards,
> > > > Gyewon Lee
> > > >
> > >
> >
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
Hi John,
The exception occurs after the evaluator spawns. I think you have a point:
Reef may be working properly, but there may be other problems with the
application.
For now I got around the issue by installing HDFS and switching the file
system to that. The directory "/tmp" is globally visible on HDFS, so
everything worked there. It's a shame I couldn't make it work on the
Lustre file system (in which /tmp is *not* mounted), but I think this is an
application problem instead of a Reef problem.
Cheers, Dylan
On Wed, Aug 3, 2016 at 11:28 PM, John Yang <jo...@gmail.com> wrote:
> Hi Dylan,
>
> Does the exception occur before the Evaluator process is spawned or after?
> If it's before, then I think YARN is failing to locate the directory.
> If it's after, I really don't have a good idea as the parameter seem to be
> for Driver only, being in the package "org.apache.reef.driver.parameters".
>
> Thanks,
> John
>
>
> On Wed, Aug 3, 2016 at 4:55 AM, Dylan Hutchison <
> dhutchis@cs.washington.edu>
> wrote:
>
> > Thanks Gyewon and John. I made some progress: the driver now sets the
> > DriverJobSubmissionDirectory
> > <
> > https://reef.apache.org/apidocs/0.15.0/org/apache/
> reef/javabridge/generic/Launch.DriverJobSubmissionDirectory.html
> > >
> > to the appropriate value and the Application Master recognizes it.
> >
> > Unfortunately, this information does not propagate to the containers that
> > the Driver requests, when it requests an Evaluator. The containers
> > continue to use the default value "/tmp". The message the driver
> receives
> > from the container for an allocated evaluator is a FileNotFoundException,
> > because the evaluator tries to use the wrong directory and can't find the
> > jars that Reef is trying to send to the evaluators.
> >
> > Is there a way to propogate the DriverJobSubmissionDirectory globally to
> > all Yarn processes? I've looked at the EvaluatorRequest
> > <
> > https://reef.apache.org/apidocs/0.15.0/org/apache/reef/driver/evaluator/
> EvaluatorRequest.html
> > >
> > API but can't find anything.
> >
> > Thanks, Dylan
> >
> >
> > On Mon, Aug 1, 2016 at 11:40 PM, Gyewon Lee <st...@gmail.com>
> wrote:
> >
> > > Hi, Dylan.
> > >
> > > In Tang NamedParameter, "default_value" is ignored when you explicitly
> > set
> > > your own value via JavaConfigurationBuilder (using methods like
> > > bindNamedParameter()) or ConfigurationModule (using method 'set()' in
> > > John's answer). So, you don't need to modify REEF source code for
> > changing
> > > default_value. In your case, John's solution will work quite nicely
> > because
> > > there is a predefined ConfigurationModule
> (YarnDriverConfiguration.CONF).
> > >
> > > In other cases where the ConfigurationModule is not set for your case,
> > then
> > > you can bind the value for NamedParameters, like
> > >
> > > JavaConfigurationBuilder cb =
> > > Tang.Factory.getTang().newConfigurationBuilder();
> > > cb.bindNamedParameter(JobSubmissionDirectoryPrefix.class,
> > > "/path/to/job/submission/directory/prefix");
> > > existingConfiguration.merge(cb.build());
> > >
> > > You don't need to do like that in this case, but I hope this would help
> > in
> > > your future REEF programming.
> > >
> > > Best regards,
> > > Gyewon Lee
> > >
> >
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by John Yang <jo...@gmail.com>.
Hi Dylan,
Does the exception occur before the Evaluator process is spawned or after?
If it's before, then I think YARN is failing to locate the directory.
If it's after, I really don't have a good idea as the parameter seem to be
for Driver only, being in the package "org.apache.reef.driver.parameters".
Thanks,
John
On Wed, Aug 3, 2016 at 4:55 AM, Dylan Hutchison <dh...@cs.washington.edu>
wrote:
> Thanks Gyewon and John. I made some progress: the driver now sets the
> DriverJobSubmissionDirectory
> <
> https://reef.apache.org/apidocs/0.15.0/org/apache/reef/javabridge/generic/Launch.DriverJobSubmissionDirectory.html
> >
> to the appropriate value and the Application Master recognizes it.
>
> Unfortunately, this information does not propagate to the containers that
> the Driver requests, when it requests an Evaluator. The containers
> continue to use the default value "/tmp". The message the driver receives
> from the container for an allocated evaluator is a FileNotFoundException,
> because the evaluator tries to use the wrong directory and can't find the
> jars that Reef is trying to send to the evaluators.
>
> Is there a way to propogate the DriverJobSubmissionDirectory globally to
> all Yarn processes? I've looked at the EvaluatorRequest
> <
> https://reef.apache.org/apidocs/0.15.0/org/apache/reef/driver/evaluator/EvaluatorRequest.html
> >
> API but can't find anything.
>
> Thanks, Dylan
>
>
> On Mon, Aug 1, 2016 at 11:40 PM, Gyewon Lee <st...@gmail.com> wrote:
>
> > Hi, Dylan.
> >
> > In Tang NamedParameter, "default_value" is ignored when you explicitly
> set
> > your own value via JavaConfigurationBuilder (using methods like
> > bindNamedParameter()) or ConfigurationModule (using method 'set()' in
> > John's answer). So, you don't need to modify REEF source code for
> changing
> > default_value. In your case, John's solution will work quite nicely
> because
> > there is a predefined ConfigurationModule (YarnDriverConfiguration.CONF).
> >
> > In other cases where the ConfigurationModule is not set for your case,
> then
> > you can bind the value for NamedParameters, like
> >
> > JavaConfigurationBuilder cb =
> > Tang.Factory.getTang().newConfigurationBuilder();
> > cb.bindNamedParameter(JobSubmissionDirectoryPrefix.class,
> > "/path/to/job/submission/directory/prefix");
> > existingConfiguration.merge(cb.build());
> >
> > You don't need to do like that in this case, but I hope this would help
> in
> > your future REEF programming.
> >
> > Best regards,
> > Gyewon Lee
> >
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
Thanks Gyewon and John. I made some progress: the driver now sets the
DriverJobSubmissionDirectory
<https://reef.apache.org/apidocs/0.15.0/org/apache/reef/javabridge/generic/Launch.DriverJobSubmissionDirectory.html>
to the appropriate value and the Application Master recognizes it.
Unfortunately, this information does not propagate to the containers that
the Driver requests, when it requests an Evaluator. The containers
continue to use the default value "/tmp". The message the driver receives
from the container for an allocated evaluator is a FileNotFoundException,
because the evaluator tries to use the wrong directory and can't find the
jars that Reef is trying to send to the evaluators.
Is there a way to propogate the DriverJobSubmissionDirectory globally to
all Yarn processes? I've looked at the EvaluatorRequest
<https://reef.apache.org/apidocs/0.15.0/org/apache/reef/driver/evaluator/EvaluatorRequest.html>
API but can't find anything.
Thanks, Dylan
On Mon, Aug 1, 2016 at 11:40 PM, Gyewon Lee <st...@gmail.com> wrote:
> Hi, Dylan.
>
> In Tang NamedParameter, "default_value" is ignored when you explicitly set
> your own value via JavaConfigurationBuilder (using methods like
> bindNamedParameter()) or ConfigurationModule (using method 'set()' in
> John's answer). So, you don't need to modify REEF source code for changing
> default_value. In your case, John's solution will work quite nicely because
> there is a predefined ConfigurationModule (YarnDriverConfiguration.CONF).
>
> In other cases where the ConfigurationModule is not set for your case, then
> you can bind the value for NamedParameters, like
>
> JavaConfigurationBuilder cb =
> Tang.Factory.getTang().newConfigurationBuilder();
> cb.bindNamedParameter(JobSubmissionDirectoryPrefix.class,
> "/path/to/job/submission/directory/prefix");
> existingConfiguration.merge(cb.build());
>
> You don't need to do like that in this case, but I hope this would help in
> your future REEF programming.
>
> Best regards,
> Gyewon Lee
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by Gyewon Lee <st...@gmail.com>.
Hi, Dylan.
In Tang NamedParameter, "default_value" is ignored when you explicitly set
your own value via JavaConfigurationBuilder (using methods like
bindNamedParameter()) or ConfigurationModule (using method 'set()' in
John's answer). So, you don't need to modify REEF source code for changing
default_value. In your case, John's solution will work quite nicely because
there is a predefined ConfigurationModule (YarnDriverConfiguration.CONF).
In other cases where the ConfigurationModule is not set for your case, then
you can bind the value for NamedParameters, like
JavaConfigurationBuilder cb =
Tang.Factory.getTang().newConfigurationBuilder();
cb.bindNamedParameter(JobSubmissionDirectoryPrefix.class,
"/path/to/job/submission/directory/prefix");
existingConfiguration.merge(cb.build());
You don't need to do like that in this case, but I hope this would help in
your future REEF programming.
Best regards,
Gyewon Lee
Re: Tang in Yarn - overriding a NamedParameter
Posted by John Yang <jo...@gmail.com>.
I think you can "set" the variable, without having to edit the source code,
just as you would do with DriverConfiguration.CONF. I guess you could do
something like:
*final Configuration yarnDriverConfiguration =
YarnDriverConfiguration.CONF *
* .set(YarnDriverConfiguration.JOB_SUBMISSION_DIRECTORY_PREFIX,
"/mydir") *
* .build(); *
Also, YarnDriverConfiguration is documented as "Additional YARN-Specific
configuration options to be merged with DriverConfiguration.", so I guess
you should merge it with DriverConfiguration like the following before
submitting the configuration to DriverLauncher.
*final Configuration driverConfWithCustomDirectoryPrefix =
Configurations.merge(yarnDriverConfiguration, driverConfiguration);*
Hope this helps.
Thanks,
John
On Tue, Aug 2, 2016 at 2:56 PM, Dylan Hutchison <dh...@cs.washington.edu>
wrote:
> Thanks John. Do you mean to edit the source code of Reef, or is there
> another way to edit the CONF variable in YarnDriverConfiguration?
>
> On Aug 1, 2016 10:03 PM, "John Yang" <jo...@gmail.com> wrote:
>
> > Hi Dylan,
> >
> > I haven't tried it myself, but maybe you can try setting the variable
> > JOB_SUBMISSION_DIRECTORY_PREFIX
> > in org.apache.reef.runtime.yarn.client.YarnDriverConfiguration.CONF, as
> the
> > variable seems to bind to JobSubmissionDirectoryPrefix.
> >
> > Thanks,
> > John
> >
> > On Tue, Aug 2, 2016 at 11:45 AM, Dylan Hutchison <
> > dhutchis@cs.washington.edu
> > > wrote:
> >
> > > Hi Reefies,
> > >
> > > I have a Reef application that runs on Yarn. Reef is responsible for
> > > distributing some jars to the various containers that start in the Yarn
> > > cluster.
> > >
> > > From what I can tell, Reef uses a JobUploader to put the jars in a
> > location
> > > visible to all the containers. I need to change this location from the
> > > default.
> > >
> > > The default location is controlled by
> > >
> > > @NamedParameter(doc = "The job submission directory prefix.",
> > > default_value = "/tmp")
> > > public final class JobSubmissionDirectoryPrefix implements
> Name<String> {
> > > }
> > >
> > > I would like to change this to a custom directory, because /tmp is not
> a
> > > globally visible location in my cluster.
> > >
> > > How do I do so?
> > >
> > > Thanks,
> > > Dylan Hutchison
> > >
> >
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by Dylan Hutchison <dh...@cs.washington.edu>.
Thanks John. Do you mean to edit the source code of Reef, or is there
another way to edit the CONF variable in YarnDriverConfiguration?
On Aug 1, 2016 10:03 PM, "John Yang" <jo...@gmail.com> wrote:
> Hi Dylan,
>
> I haven't tried it myself, but maybe you can try setting the variable
> JOB_SUBMISSION_DIRECTORY_PREFIX
> in org.apache.reef.runtime.yarn.client.YarnDriverConfiguration.CONF, as the
> variable seems to bind to JobSubmissionDirectoryPrefix.
>
> Thanks,
> John
>
> On Tue, Aug 2, 2016 at 11:45 AM, Dylan Hutchison <
> dhutchis@cs.washington.edu
> > wrote:
>
> > Hi Reefies,
> >
> > I have a Reef application that runs on Yarn. Reef is responsible for
> > distributing some jars to the various containers that start in the Yarn
> > cluster.
> >
> > From what I can tell, Reef uses a JobUploader to put the jars in a
> location
> > visible to all the containers. I need to change this location from the
> > default.
> >
> > The default location is controlled by
> >
> > @NamedParameter(doc = "The job submission directory prefix.",
> > default_value = "/tmp")
> > public final class JobSubmissionDirectoryPrefix implements Name<String> {
> > }
> >
> > I would like to change this to a custom directory, because /tmp is not a
> > globally visible location in my cluster.
> >
> > How do I do so?
> >
> > Thanks,
> > Dylan Hutchison
> >
>
Re: Tang in Yarn - overriding a NamedParameter
Posted by John Yang <jo...@gmail.com>.
Hi Dylan,
I haven't tried it myself, but maybe you can try setting the variable
JOB_SUBMISSION_DIRECTORY_PREFIX
in org.apache.reef.runtime.yarn.client.YarnDriverConfiguration.CONF, as the
variable seems to bind to JobSubmissionDirectoryPrefix.
Thanks,
John
On Tue, Aug 2, 2016 at 11:45 AM, Dylan Hutchison <dhutchis@cs.washington.edu
> wrote:
> Hi Reefies,
>
> I have a Reef application that runs on Yarn. Reef is responsible for
> distributing some jars to the various containers that start in the Yarn
> cluster.
>
> From what I can tell, Reef uses a JobUploader to put the jars in a location
> visible to all the containers. I need to change this location from the
> default.
>
> The default location is controlled by
>
> @NamedParameter(doc = "The job submission directory prefix.",
> default_value = "/tmp")
> public final class JobSubmissionDirectoryPrefix implements Name<String> {
> }
>
> I would like to change this to a custom directory, because /tmp is not a
> globally visible location in my cluster.
>
> How do I do so?
>
> Thanks,
> Dylan Hutchison
>