You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Yao Wang <ya...@ozstrategy.com> on 2016/06/06 08:05:37 UTC
how to stop the mesos executor process in JVM?
Hi , all !
I write my own executor to run code,
I override the launchTask method like that :
------------------------------------------------------------------------------------------------------------------------------------------------
@Override public void launchTask(ExecutorDriver driver, Protos.TaskInfo task) {
LOGGER.info("Executor is launching task#{}\n...", task);
//before launch
driver.sendStatusUpdate(
Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(
Protos.TaskState.TASK_RUNNING).build());
LOGGER.info("Add your bussiness code hear .. ");
//bussiness code hear
//after launch
driver.sendStatusUpdate(
Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(Protos.TaskState.TASK_FINISHED).setData(
ByteString.copyFromUtf8(
"${taksData}")).build());
} // end method launchTask
------------------------------------------------------------------------------------------------------------------------------------------------
And i build the commandInfo like that:
------------------------------------------------------------------------------------------------------------------------------------------------
String executorCommand = String.format("java -jar %s", extractPath(executorJarPath));
Protos.CommandInfo.URI.Builder executorJarURI = Protos.CommandInfo.URI.newBuilder().setValue(executorJarPath); // executorJarURI is local uri or hadoop
Protos.CommandInfo.Builder commandInfoBuilder = Protos.CommandInfo.newBuilder().setEnvironment(envBuilder).setValue(
executorCommand).addUris(executorJarURI); // executorJarURI is local uri or hadoop
long ctms = System.nanoTime();
Protos.ExecutorID.Builder executorIDBuilder = Protos.ExecutorID.newBuilder().setValue(new StringBuilder().append(
ctms).append("-").append(task.getTaskRequestId()).toString());
Protos.ExecutorInfo.Builder executorInfoBuilder = Protos.ExecutorInfo.newBuilder().setExecutorId(
executorIDBuilder).setCommand(commandInfoBuilder).setName("flexcloud-executor-2.0.1-" + ctms).setSource("java");
// TaskInfo
Protos.TaskInfo.Builder taskInfoBuilder = Protos.TaskInfo.newBuilder().setName(task.getTaskName()).setTaskId(
taskIDBuilder).setSlaveId(offer.getSlaveId()).setExecutor(executorInfoBuilder);
return taskInfoBuilder.build();
------------------------------------------------------------------------------------------------------------------------------------------------
After run the executor with mesos for several times , i found every executor was not exit ,
I execute $ ps -ef | grep “java -jar” on the slave machine , that shows me :
wangyao$ ps -ef | grep "java -jar"
501 20078 19302 0 3:54下午 ?? 0:15.77 /usr/bin/java -jar flexcloud-executor.jar
501 20154 19302 0 3:54下午 ?? 0:17.92 /usr/bin/java -jar flexcloud-executor.jar
501 20230 19302 0 3:54下午 ?? 0:16.13 /usr/bin/java -jar flexcloud-executor.jar
In order to stop these process after running a executor, first , i tried to add code "driver.stop()” or “driver.abort()” to the Executor’s launchTask method, but it is unused.
So, I add code “System.exit(0)” , stop the JVM directly……. it works …
I have doubt about this way to stop executor , it is the only way to do that?
Re: how to stop the mesos executor process in JVM?
Posted by Sharma Podila <sp...@netflix.com>.
Yao, in our Java executor, we explicitly call System.exit(0) after we have
successfully sent the last finished message. However, note that there can
be a little bit of a timing issue here. Once we send the last message, we
call an asynchronous "sleep some and exit" routine. This gives the mesos
driver a chance to send the last message successfully before the executor
JVM exits. Usually, a sleep of 2-3 secs should suffice. There may be a more
elegant way to handle this timing issue, but, I haven't looked at it
recently.
On Mon, Jun 6, 2016 at 6:34 AM, Vinod Kone <vi...@apache.org> wrote:
> Couple things.
>
> You need to do the business logic and status update sending in a different
> thread than synchronously in launchTask(). This is because the driver
> doesn't send messages to the agent unless the launchTask() method returns.
> See
> https://github.com/apache/mesos/blob/master/src/examples/java/TestExecutor.java
> for example.
>
> Regarding exiting the executor, driver.stop() or driver.abort() only stops
> the driver, i.e., your executor won't be able to send or receive messages
> from the agent. It is entirely up to the executor process to exit.
>
> HTH,
>
>
>
> On Mon, Jun 6, 2016 at 4:05 AM, Yao Wang <ya...@ozstrategy.com> wrote:
>
>> Hi , all !
>>
>> I write my own executor to run code,
>>
>> I override the launchTask method like that :
>>
>>
>> ------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> @Override public void launchTask(ExecutorDriver driver, Protos.TaskInfo
>> task) {
>> LOGGER.info("Executor is launching task#{}\n...", task);
>> //before launch
>> driver.sendStatusUpdate(
>>
>> Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(
>> Protos.TaskState.TASK_RUNNING).build());
>>
>> LOGGER.info("Add your bussiness code hear .. ");
>> //bussiness code hear
>>
>>
>> //after launch
>> driver.sendStatusUpdate(
>>
>> Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(Protos.TaskState.TASK_FINISHED).setData(
>> ByteString.copyFromUtf8(
>> "${taksData}")).build());
>>
>>
>> } // end method launchTask
>> ------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> And i build the commandInfo like that:
>>
>> ------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> String executorCommand = String.format("java -jar %s",
>> extractPath(executorJarPath));
>>
>> Protos.CommandInfo.URI.Builder executorJarURI =
>> Protos.CommandInfo.URI.newBuilder().setValue(executorJarPath); //
>> executorJarURI is local uri or hadoop
>>
>> Protos.CommandInfo.Builder commandInfoBuilder =
>> Protos.CommandInfo.newBuilder().setEnvironment(envBuilder).setValue(
>> executorCommand).addUris(executorJarURI); // executorJarURI is
>> local uri or hadoop
>>
>> long ctms = System.nanoTime();
>>
>> Protos.ExecutorID.Builder executorIDBuilder =
>> Protos.ExecutorID.newBuilder().setValue(new StringBuilder().append(
>> ctms).append("-").append(task.getTaskRequestId()).toString());
>> Protos.ExecutorInfo.Builder executorInfoBuilder =
>> Protos.ExecutorInfo.newBuilder().setExecutorId(
>>
>> executorIDBuilder).setCommand(commandInfoBuilder).setName("flexcloud-executor-2.0.1-"
>> + ctms).setSource("java");
>>
>> // TaskInfo
>> Protos.TaskInfo.Builder taskInfoBuilder =
>> Protos.TaskInfo.newBuilder().setName(task.getTaskName()).setTaskId(
>>
>> taskIDBuilder).setSlaveId(offer.getSlaveId()).setExecutor(executorInfoBuilder);
>>
>>
>> return taskInfoBuilder.build();
>> ------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> After run the executor with mesos for several times , i found every
>> executor was not exit ,
>>
>> I execute $ ps -ef | grep “java -jar” on the slave machine , that
>> shows me :
>>
>> wangyao$ ps -ef | grep "java -jar"
>> 501 20078 19302 0 3:54下午 ?? 0:15.77 /usr/bin/java -jar
>> flexcloud-executor.jar
>> 501 20154 19302 0 3:54下午 ?? 0:17.92 /usr/bin/java -jar
>> flexcloud-executor.jar
>> 501 20230 19302 0 3:54下午 ?? 0:16.13 /usr/bin/java -jar
>> flexcloud-executor.jar
>>
>> In order to stop these process after running a executor, first , i
>> tried to add code "driver.stop()” or “driver.abort()” to the Executor’s
>> launchTask method, but it is unused.
>> So, I add code “System.exit(0)” , stop the JVM directly……. it works …
>>
>> I have doubt about this way to stop executor , it is the only way to do
>> that?
>>
>>
>
Re: how to stop the mesos executor process in JVM?
Posted by Vinod Kone <vi...@apache.org>.
Couple things.
You need to do the business logic and status update sending in a different
thread than synchronously in launchTask(). This is because the driver
doesn't send messages to the agent unless the launchTask() method returns.
See
https://github.com/apache/mesos/blob/master/src/examples/java/TestExecutor.java
for example.
Regarding exiting the executor, driver.stop() or driver.abort() only stops
the driver, i.e., your executor won't be able to send or receive messages
from the agent. It is entirely up to the executor process to exit.
HTH,
On Mon, Jun 6, 2016 at 4:05 AM, Yao Wang <ya...@ozstrategy.com> wrote:
> Hi , all !
>
> I write my own executor to run code,
>
> I override the launchTask method like that :
>
>
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> @Override public void launchTask(ExecutorDriver driver, Protos.TaskInfo
> task) {
> LOGGER.info("Executor is launching task#{}\n...", task);
> //before launch
> driver.sendStatusUpdate(
>
> Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(
> Protos.TaskState.TASK_RUNNING).build());
>
> LOGGER.info("Add your bussiness code hear .. ");
> //bussiness code hear
>
>
> //after launch
> driver.sendStatusUpdate(
>
> Protos.TaskStatus.newBuilder().setTaskId(task.getTaskId()).setState(Protos.TaskState.TASK_FINISHED).setData(
> ByteString.copyFromUtf8(
> "${taksData}")).build());
>
>
> } // end method launchTask
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> And i build the commandInfo like that:
>
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> String executorCommand = String.format("java -jar %s",
> extractPath(executorJarPath));
>
> Protos.CommandInfo.URI.Builder executorJarURI =
> Protos.CommandInfo.URI.newBuilder().setValue(executorJarPath); //
> executorJarURI is local uri or hadoop
>
> Protos.CommandInfo.Builder commandInfoBuilder =
> Protos.CommandInfo.newBuilder().setEnvironment(envBuilder).setValue(
> executorCommand).addUris(executorJarURI); // executorJarURI is
> local uri or hadoop
>
> long ctms = System.nanoTime();
>
> Protos.ExecutorID.Builder executorIDBuilder =
> Protos.ExecutorID.newBuilder().setValue(new StringBuilder().append(
> ctms).append("-").append(task.getTaskRequestId()).toString());
> Protos.ExecutorInfo.Builder executorInfoBuilder =
> Protos.ExecutorInfo.newBuilder().setExecutorId(
>
> executorIDBuilder).setCommand(commandInfoBuilder).setName("flexcloud-executor-2.0.1-"
> + ctms).setSource("java");
>
> // TaskInfo
> Protos.TaskInfo.Builder taskInfoBuilder =
> Protos.TaskInfo.newBuilder().setName(task.getTaskName()).setTaskId(
>
> taskIDBuilder).setSlaveId(offer.getSlaveId()).setExecutor(executorInfoBuilder);
>
>
> return taskInfoBuilder.build();
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> After run the executor with mesos for several times , i found every
> executor was not exit ,
>
> I execute $ ps -ef | grep “java -jar” on the slave machine , that shows
> me :
>
> wangyao$ ps -ef | grep "java -jar"
> 501 20078 19302 0 3:54下午 ?? 0:15.77 /usr/bin/java -jar
> flexcloud-executor.jar
> 501 20154 19302 0 3:54下午 ?? 0:17.92 /usr/bin/java -jar
> flexcloud-executor.jar
> 501 20230 19302 0 3:54下午 ?? 0:16.13 /usr/bin/java -jar
> flexcloud-executor.jar
>
> In order to stop these process after running a executor, first , i
> tried to add code "driver.stop()” or “driver.abort()” to the Executor’s
> launchTask method, but it is unused.
> So, I add code “System.exit(0)” , stop the JVM directly……. it works …
>
> I have doubt about this way to stop executor , it is the only way to do
> that?
>
>