Evaluating Helix as the task execution framework

[DImuthu Upeksha <di...@gmail.com>]

Hi Team,

Depending on the feedback received in the last meeting with Suresh and the
team, I worked on trying out Helix's task framework as the task execution
framework in Airavata. After going through the documentation and existing
work of Gaurav, I came up with a design [1] that we can deploy Helix's task
framework as the task execution engine of Airavata in a containerized
environment.


​

In addition to that, I created a graphical workflow composer that can
develop and deploy workflows into the engine. This was designed as
explained in [2].


​
You can have a look at the screen cast that demonstrates an end to end
workflow execution from this [3] video.


*Summary*

1. Although Helix is not the ideal candidate for container orchestration, I
found Helix is well fitted for workflow execution in Airavata instead of
building a task execution framework from the beginning.

2. All the Helix agents (controllers and participants) can be wrapped as
the Docker containers and orchestrate on Kubernetes or DC/OS for high
availability and scalability.

3. With the APIs of Helix, we can easily create and stop workflows at any
time and the burden of managing those states is taken cared by Helix. This
makes our life easy and let us focus more on improving and stabalizing the
framework around that.

*Note*

Currently above demonstration is run inside the IDE and technically it is
possible to dockerize each component and deploy into a Kubernetes cluster.
I will perform the final packaging once the design is evaluated by the
team. You can find the code up to that point from here [4]

Please share your views and suggestions.

[1]
https://docs.google.com/document/d/1P47nZU82YhHm2mC1PfonKo8kMjD_kVf0GeYQ0NZpI_Q/edit?usp=sharing
​
[2]
https://docs.google.com/document/d/118dNH6eycFdzReeNM8CzsqFgt67isohIHmv08pMhCPM/edit?usp=sharing
[3] https://youtu.be/7ptjyBdj2dQ
[4]
https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-kubernetes

Thanks
Dimuthu

Re: Evaluating Helix as the task execution framework

[DImuthu Upeksha <di...@gmail.com>]

Hi all

Adding helix-dev as this is mainly focused on deploying Helix Tasks in a
containerized environment.

To provide a general overview, I'm implementing a task execution framework
for Airavata using the task framework of Helix. I'm using Helix tasks as
the constructs that synthesize a workflow of Airavata. Example for possible
tasks are executing a scientific job on a remote resource, pushing input
files to scientific jobs and fetching outputs once a job is completed.

Once I came up with a working prototype [1], my next consideration was
about the deployment aspect of these tasks. I chose to package these tasks
as docker containers and deploy in a Kubernetes cluster. Helix Task
components were packaged as *one task per participant and one participant
per container* (pod). By doing that, I had to specify an instance group tag
for each participant (container) in order to find correct participant when
scheduling a task by Helix controller.

Diagram:
https://drive.google.com/file/d/1P1JGgmzx7yttygpxnZhuApbJMzMlafs5/view?usp=sharing
​
This enables us to independently scale and update each task type without
interrupting other tasks in the framework. If we want to scale a particular
task type, we can command to Kubernetes API with a replication count and
Kubernetes brings up the number of replications of the task to the
specified level. Next problem is, what do we provide as the names of the
participants which are deployed as containers. I used the host name of the
pod which is unique along the Kubernetes cluster. This makes sure that two
participants with the same name do not exist.

You can find Kubernetes scripts that perform above mentioned deployment
from here [2] and it will be really helpful if @heix-dev can provide
feedback or the best practices upon the above approach of containerizing
Helix tasks and participants.

[1] https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-
kubernetes/modules/microservices/tasks
[2] https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-
kubernetes/scripts/k8s

Thanks

Re: Evaluating Helix as the task execution framework

[DImuthu Upeksha <di...@gmail.com>]

Hi all

Adding helix-dev as this is mainly focused on deploying Helix Tasks in a
containerized environment.

To provide a general overview, I'm implementing a task execution framework
for Airavata using the task framework of Helix. I'm using Helix tasks as
the constructs that synthesize a workflow of Airavata. Example for possible
tasks are executing a scientific job on a remote resource, pushing input
files to scientific jobs and fetching outputs once a job is completed.

Once I came up with a working prototype [1], my next consideration was
about the deployment aspect of these tasks. I chose to package these tasks
as docker containers and deploy in a Kubernetes cluster. Helix Task
components were packaged as *one task per participant and one participant
per container* (pod). By doing that, I had to specify an instance group tag
for each participant (container) in order to find correct participant when
scheduling a task by Helix controller.


​
This enables us to independently scale and update each task type without
interrupting other tasks in the framework. If we want to scale a particular
task type, we can command to Kubernetes API with a replication count and
Kubernetes brings up the number of replications of the task to the
specified level. Next problem is, what do we provide as the names of the
participants which are deployed as containers. I used the host name of the
pod which is unique along the Kubernetes cluster. This makes sure that two
participants with the same name do not exist.

You can find Kubernetes scripts that perform above mentioned deployment
from here [2] and it will be really helpful if @heix-dev can provide
feedback or the best practices upon the above approach of containerizing
Helix tasks and participants.

[1]
https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-kubernetes/modules/microservices/tasks
[2]
https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-kubernetes/scripts/k8s

Thanks
Dimuthu

On Wed, Nov 22, 2017 at 12:44 AM, DImuthu Upeksha <
dimuthu.upeksha2@gmail.com> wrote:

> Hi Supun,
>
> Thanks for the comments and plese find my response inline
>
>
> On 21 Nov 2017 10:17, "Supun Nakandala" <su...@gmail.com> wrote:
>
> Hi Dimuthu,
>
> I think this is great progress.
>
> I would like to know whether you have already considered about the failure
> scenarios too?
>
> Specifically, I am curious about how you handle
>  1. Task fails. This can happen due to several reasons. 1. invalid
> configurations(e.g. SSH keys) and inputs 2. downstream system issues such
> as network and file system failures 3. task fails due to other reasons(OOM,
> OS kills, etc..)
>
>
> Actually we can categorize them in to two main failure types.
> 1. Re try-able failures
> 2. Non re try-able failures
>
> If a task fails due to a temporary network failure, implementation of the
> task should return a Fail state.
> return new TaskResult(TaskResult.Status.FAILED, "Task failed");
> Then Helix controller can retry this task up to a configurable iterations
>
>  If the failure is not a recoverable one like SSH key mismatch, then we
> can return a Fatal Failure state
> return new TaskResult(TaskResult.Status.FATAL_FAILED, "Task failed");
> Then Helix controller immediately stops the workflow
>
> OOM and OS kills should be handled in a different way. As we are deploying
> a single task in a single participant, if an OOM error comes, there is no
> use of having a Participant. So in case of an error, we should immediately
> kill the participant by calling System.exit(0). Further steps are covered
> in the next answer.
>
>
>  2. What will happen if a Helix participant fails?
>
>
> This design assumes that we are deploying Participants as containers in a
> Kubernetes cluster. Using Kubernetes, we can easily keep a fault tolerant
> pool of a specific set of Participants. If one participant fails, another
> one will be automatically created. However the problem is, what happens if
> the participant (task) fails while in the middle on an execution? If the
> participant hasn't published any state, then Helix will start the execution
> of that task in another participant of the pool. So it is the responsible
> of the developer to write critical tasks in a transactional or recoverable
> manner.
>
>
>  3. What will happen if one of the scheduler or event sink fails while
> processing a request?
>
>
> Good question. Event sink failures are already handled. Event sink only
> acknowledges to kafka topic if it can successfully publish the read state
> to API. If the Sink fails in between, a new Event sink will start to read
> from last acknowledged position of the topic. However we can't do this
> approach to Scheduler as scheduler might take a significant amount of time
> to finish a workflow and if we don't acknowledge to the topic, other
> workflow launch commands will be blocked in the topic. My suggestion is to
> keep an ephemeral node in zookeeper for each running workflow id. Owner of
> that node is the scheduler instance which launches the workflow to Helix.
> If the scheduler got killed while monitoring the workflow, new scheduler
> should acquire the released ephemeral node and continue monitoring the
> workflow. To do this, we might need to have a scheduled loop inside each
> scheduler to monitor unfinished workflows. Please do let me know if you
> have an alternative approach
>
>
> Also, it's not clear to me where will the workflow execution logic will
> run. I see that the feedback loop of events is going back to the API server
> and not to the schedulers. Shouldn't the scheduler get this information
> back (for example if a task fails it will have to resubmit)?
>
>
> Pease note that these events are specific to our domain logic and do not
> confuse with the states (FAILED, FATAL_FAILED) that i have mentioned
> earlier which are specific to Helix. My initial idea also was to get the
> feedback of task events to the scheduler.  However the problem is, once a
> workflow (task dag) is submitted to Helix, there is no control for
> Scheduler to handle tasks in the Workflow. It is being done by the Helix
> Controller. Only thing Scheduler can do is to stop the workflow. So I would
> suggest, if you want to retry a task, simply return a FAILED state as
> mentioned above and Helix controller will be automatically retry the task.
> Please let me know if you have use cases where we want to have more
> specific retry logics. If there is any, we might have to think of using a
> Job Queue as mentioned in [1] but this will introduce lot of complexities
> as the scheduling logic completely comes into our hand.
>
> [1] http://helix.apache.org/0.6.9-docs/tutorial_task_framework.html
>
>
> Thank you.
>
> On Mon, Nov 20, 2017 at 1:06 AM, DImuthu Upeksha <
> dimuthu.upeksha2@gmail.com> wrote:
>
>> Hi Team,
>>
>> Depending on the feedback received in the last meeting with Suresh and
>> the team, I worked on trying out Helix's task framework as the task
>> execution framework in Airavata. After going through the documentation and
>> existing work of Gaurav, I came up with a design [1] that we can deploy
>> Helix's task framework as the task execution engine of Airavata in a
>> containerized environment.
>>
>>
>> ​
>>
>> In addition to that, I created a graphical workflow composer that can
>> develop and deploy workflows into the engine. This was designed as
>> explained in [2].
>>
>>
>> ​
>> You can have a look at the screen cast that demonstrates an end to end
>> workflow execution from this [3] video.
>>
>>
>> *Summary*
>>
>> 1. Although Helix is not the ideal candidate for container orchestration,
>> I found Helix is well fitted for workflow execution in Airavata instead of
>> building a task execution framework from the beginning.
>>
>> 2. All the Helix agents (controllers and participants) can be wrapped as
>> the Docker containers and orchestrate on Kubernetes or DC/OS for high
>> availability and scalability.
>>
>> 3. With the APIs of Helix, we can easily create and stop workflows at any
>> time and the burden of managing those states is taken cared by Helix. This
>> makes our life easy and let us focus more on improving and stabalizing the
>> framework around that.
>>
>> *Note*
>>
>> Currently above demonstration is run inside the IDE and technically it is
>> possible to dockerize each component and deploy into a Kubernetes cluster.
>> I will perform the final packaging once the design is evaluated by the
>> team. You can find the code up to that point from here [4]
>>
>> Please share your views and suggestions.
>>
>> [1] https://docs.google.com/document/d/1P47nZU82YhHm2mC1PfonKo8k
>> MjD_kVf0GeYQ0NZpI_Q/edit?usp=sharing​
>> [2] https://docs.google.com/document/d/118dNH6eycFdzReeNM8Cz
>> sqFgt67isohIHmv08pMhCPM/edit?usp=sharing
>> [3] https://youtu.be/7ptjyBdj2dQ
>> [4] https://github.com/DImuthuUpe/airavata-sandbox/tree/mast
>> er/airavata-kubernetes
>>
>> Thanks
>> Dimuthu
>>
>
>
>

Re: Evaluating Helix as the task execution framework

[DImuthu Upeksha <di...@gmail.com>]

Hi Supun,

Thanks for the comments and plese find my response inline


On 21 Nov 2017 10:17, "Supun Nakandala" <su...@gmail.com> wrote:

Hi Dimuthu,

I think this is great progress.

I would like to know whether you have already considered about the failure
scenarios too?

Specifically, I am curious about how you handle
 1. Task fails. This can happen due to several reasons. 1. invalid
configurations(e.g. SSH keys) and inputs 2. downstream system issues such
as network and file system failures 3. task fails due to other reasons(OOM,
OS kills, etc..)


Actually we can categorize them in to two main failure types.
1. Re try-able failures
2. Non re try-able failures

If a task fails due to a temporary network failure, implementation of the
task should return a Fail state.
return new TaskResult(TaskResult.Status.FAILED, "Task failed");
Then Helix controller can retry this task up to a configurable iterations

 If the failure is not a recoverable one like SSH key mismatch, then we can
return a Fatal Failure state
return new TaskResult(TaskResult.Status.FATAL_FAILED, "Task failed");
Then Helix controller immediately stops the workflow

OOM and OS kills should be handled in a different way. As we are deploying
a single task in a single participant, if an OOM error comes, there is no
use of having a Participant. So in case of an error, we should immediately
kill the participant by calling System.exit(0). Further steps are covered
in the next answer.


 2. What will happen if a Helix participant fails?


This design assumes that we are deploying Participants as containers in a
Kubernetes cluster. Using Kubernetes, we can easily keep a fault tolerant
pool of a specific set of Participants. If one participant fails, another
one will be automatically created. However the problem is, what happens if
the participant (task) fails while in the middle on an execution? If the
participant hasn't published any state, then Helix will start the execution
of that task in another participant of the pool. So it is the responsible
of the developer to write critical tasks in a transactional or recoverable
manner.


 3. What will happen if one of the scheduler or event sink fails while
processing a request?


Good question. Event sink failures are already handled. Event sink only
acknowledges to kafka topic if it can successfully publish the read state
to API. If the Sink fails in between, a new Event sink will start to read
from last acknowledged position of the topic. However we can't do this
approach to Scheduler as scheduler might take a significant amount of time
to finish a workflow and if we don't acknowledge to the topic, other
workflow launch commands will be blocked in the topic. My suggestion is to
keep an ephemeral node in zookeeper for each running workflow id. Owner of
that node is the scheduler instance which launches the workflow to Helix.
If the scheduler got killed while monitoring the workflow, new scheduler
should acquire the released ephemeral node and continue monitoring the
workflow. To do this, we might need to have a scheduled loop inside each
scheduler to monitor unfinished workflows. Please do let me know if you
have an alternative approach


Also, it's not clear to me where will the workflow execution logic will
run. I see that the feedback loop of events is going back to the API server
and not to the schedulers. Shouldn't the scheduler get this information
back (for example if a task fails it will have to resubmit)?


Pease note that these events are specific to our domain logic and do not
confuse with the states (FAILED, FATAL_FAILED) that i have mentioned
earlier which are specific to Helix. My initial idea also was to get the
feedback of task events to the scheduler.  However the problem is, once a
workflow (task dag) is submitted to Helix, there is no control for
Scheduler to handle tasks in the Workflow. It is being done by the Helix
Controller. Only thing Scheduler can do is to stop the workflow. So I would
suggest, if you want to retry a task, simply return a FAILED state as
mentioned above and Helix controller will be automatically retry the task.
Please let me know if you have use cases where we want to have more
specific retry logics. If there is any, we might have to think of using a
Job Queue as mentioned in [1] but this will introduce lot of complexities
as the scheduling logic completely comes into our hand.

[1] http://helix.apache.org/0.6.9-docs/tutorial_task_framework.html


Thank you.

On Mon, Nov 20, 2017 at 1:06 AM, DImuthu Upeksha <dimuthu.upeksha2@gmail.com
> wrote:

> Hi Team,
>
> Depending on the feedback received in the last meeting with Suresh and the
> team, I worked on trying out Helix's task framework as the task execution
> framework in Airavata. After going through the documentation and existing
> work of Gaurav, I came up with a design [1] that we can deploy Helix's task
> framework as the task execution engine of Airavata in a containerized
> environment.
>
>
> ​
>
> In addition to that, I created a graphical workflow composer that can
> develop and deploy workflows into the engine. This was designed as
> explained in [2].
>
>
> ​
> You can have a look at the screen cast that demonstrates an end to end
> workflow execution from this [3] video.
>
>
> *Summary*
>
> 1. Although Helix is not the ideal candidate for container orchestration,
> I found Helix is well fitted for workflow execution in Airavata instead of
> building a task execution framework from the beginning.
>
> 2. All the Helix agents (controllers and participants) can be wrapped as
> the Docker containers and orchestrate on Kubernetes or DC/OS for high
> availability and scalability.
>
> 3. With the APIs of Helix, we can easily create and stop workflows at any
> time and the burden of managing those states is taken cared by Helix. This
> makes our life easy and let us focus more on improving and stabalizing the
> framework around that.
>
> *Note*
>
> Currently above demonstration is run inside the IDE and technically it is
> possible to dockerize each component and deploy into a Kubernetes cluster.
> I will perform the final packaging once the design is evaluated by the
> team. You can find the code up to that point from here [4]
>
> Please share your views and suggestions.
>
> [1] https://docs.google.com/document/d/1P47nZU82YhHm2mC1PfonKo8k
> MjD_kVf0GeYQ0NZpI_Q/edit?usp=sharing​
> [2] https://docs.google.com/document/d/118dNH6eycFdzReeNM8Cz
> sqFgt67isohIHmv08pMhCPM/edit?usp=sharing
> [3] https://youtu.be/7ptjyBdj2dQ
> [4] https://github.com/DImuthuUpe/airavata-sandbox/tree/mast
> er/airavata-kubernetes
>
> Thanks
> Dimuthu
>

Re: Evaluating Helix as the task execution framework

[Supun Nakandala <su...@gmail.com>]

Hi Dimuthu,

I think this is great progress.

I would like to know whether you have already considered about the failure
scenarios too?

Specifically, I am curious about how you handle
 1. Task fails. This can happen due to several reasons. 1. invalid
configurations(e.g. SSH keys) and inputs 2. downstream system issues such
as network and file system failures 3. task fails due to other reasons(OOM,
OS kills, etc..)
 2. What will happen if a Helix participant fails?
 3. What will happen if one of the scheduler or event sink fails while
processing a request?


Also, it's not clear to me where will the workflow execution logic will
run. I see that the feedback loop of events is going back to the API server
and not to the schedulers. Shouldn't the scheduler get this information
back (for example if a task fails it will have to resubmit)?

Thank you.

On Mon, Nov 20, 2017 at 1:06 AM, DImuthu Upeksha <dimuthu.upeksha2@gmail.com
> wrote:

> Hi Team,
>
> Depending on the feedback received in the last meeting with Suresh and the
> team, I worked on trying out Helix's task framework as the task execution
> framework in Airavata. After going through the documentation and existing
> work of Gaurav, I came up with a design [1] that we can deploy Helix's task
> framework as the task execution engine of Airavata in a containerized
> environment.
>
>
> ​
>
> In addition to that, I created a graphical workflow composer that can
> develop and deploy workflows into the engine. This was designed as
> explained in [2].
>
>
> ​
> You can have a look at the screen cast that demonstrates an end to end
> workflow execution from this [3] video.
>
>
> *Summary*
>
> 1. Although Helix is not the ideal candidate for container orchestration,
> I found Helix is well fitted for workflow execution in Airavata instead of
> building a task execution framework from the beginning.
>
> 2. All the Helix agents (controllers and participants) can be wrapped as
> the Docker containers and orchestrate on Kubernetes or DC/OS for high
> availability and scalability.
>
> 3. With the APIs of Helix, we can easily create and stop workflows at any
> time and the burden of managing those states is taken cared by Helix. This
> makes our life easy and let us focus more on improving and stabalizing the
> framework around that.
>
> *Note*
>
> Currently above demonstration is run inside the IDE and technically it is
> possible to dockerize each component and deploy into a Kubernetes cluster.
> I will perform the final packaging once the design is evaluated by the
> team. You can find the code up to that point from here [4]
>
> Please share your views and suggestions.
>
> [1] https://docs.google.com/document/d/1P47nZU82YhHm2mC1PfonKo8kMjD_
> kVf0GeYQ0NZpI_Q/edit?usp=sharing​
> [2] https://docs.google.com/document/d/118dNH6eycFdzReeNM8CzsqFgt67is
> ohIHmv08pMhCPM/edit?usp=sharing
> [3] https://youtu.be/7ptjyBdj2dQ
> [4] https://github.com/DImuthuUpe/airavata-sandbox/tree/master/airavata-
> kubernetes
>
> Thanks
> Dimuthu
>