[DISCUSS] Is there an opportunity work together?

[Josh Fischer <jo...@joshfischer.io>]

Hi all,

Heron is starting the process of graduating from the incubator and an idea
was brought up by one of our mentors.

Does anyone see synergies with the joining the two communities and/or
projects?

- Josh

-- 
Sent from A Mobile Device

Re: [DISCUSS] Is there an opportunity work together?

[Saad Ur Rahman <sa...@apache.org>]

Just my 2-cents on the issue:

My primary concern would be the stability of the current APIs and
interfaces. We must not break existing "customer" tooling and
workflows. I do not think this will be an issue if we maintain access
to the current URI cluster endpoints. The other issue will be if the
interface commands vary between Storm and Heron.

My consideration is architectural regarding the dispatcher; is it a
client-side tool or a server-side gateway?

On the client side, we could employ a simple tool that reads
configurations from a file, environment variables, or from the
parameters on the CLI. The tool would then dispatch to the correct
URI. This would be the most straightforward solution.

./Dispatcher <Heron | Storm> <Command> <Parameters…>

For a server-side solution, we would need a gateway service that would
dispatch to the backend. We could write our own tooling or use Nginx.
With Nginx, we could perform routing and L7 load balancing based on
the path in the URI (http://server.address/heron/... or
http://server.address/storm/...). I believe this would be the most
convenient and easy-to-maintain server-side solution.

If deployed as a tool we have built (server-side gateway) on K8, we
would employ a headless service as our L4 load balancer. We could also
employ NetworkPolicies to restrict access to the Storm and Heron
backends through the dispatcher gateway. This would, of course, break
current "customer" tooling and workflows but add security.

What do you think?

On Tue, Aug 30, 2022 at 1:50 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Sending a reply to not end the email on a “yes/no” answer.
>
> I had the not so well thought of idea of creating some sort of shared
> runner.  Could we create a service or abstraction over each of the
> different engines to run the same job?  What issues would we run into?
> What benefits could come from this?
>
> I’d like to hear others ideas on what could be as well.  What do each of
> you think?
>
> On Tue, Aug 30, 2022 at 12:45 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> > Hi all,
> >
> > Heron is starting the process of graduating from the incubator and an idea
> > was brought up by one of our mentors.
> >
> > Does anyone see synergies with the joining the two communities and/or
> > projects?
> >
> > - Josh
> >
> > --
> > Sent from A Mobile Device
> >
> --
> Sent from A Mobile Device

Re: [DISCUSS] Is there an opportunity work together?

[Saad Ur Rahman <sa...@apache.org>]

Just my 2-cents on the issue:

My primary concern would be the stability of the current APIs and
interfaces. We must not break existing "customer" tooling and
workflows. I do not think this will be an issue if we maintain access
to the current URI cluster endpoints. The other issue will be if the
interface commands vary between Storm and Heron.

My consideration is architectural regarding the dispatcher; is it a
client-side tool or a server-side gateway?

On the client side, we could employ a simple tool that reads
configurations from a file, environment variables, or from the
parameters on the CLI. The tool would then dispatch to the correct
URI. This would be the most straightforward solution.

./Dispatcher <Heron | Storm> <Command> <Parameters…>

For a server-side solution, we would need a gateway service that would
dispatch to the backend. We could write our own tooling or use Nginx.
With Nginx, we could perform routing and L7 load balancing based on
the path in the URI (http://server.address/heron/... or
http://server.address/storm/...). I believe this would be the most
convenient and easy-to-maintain server-side solution.

If deployed as a tool we have built (server-side gateway) on K8, we
would employ a headless service as our L4 load balancer. We could also
employ NetworkPolicies to restrict access to the Storm and Heron
backends through the dispatcher gateway. This would, of course, break
current "customer" tooling and workflows but add security.

What do you think?

On Tue, Aug 30, 2022 at 1:50 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Sending a reply to not end the email on a “yes/no” answer.
>
> I had the not so well thought of idea of creating some sort of shared
> runner.  Could we create a service or abstraction over each of the
> different engines to run the same job?  What issues would we run into?
> What benefits could come from this?
>
> I’d like to hear others ideas on what could be as well.  What do each of
> you think?
>
> On Tue, Aug 30, 2022 at 12:45 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> > Hi all,
> >
> > Heron is starting the process of graduating from the incubator and an idea
> > was brought up by one of our mentors.
> >
> > Does anyone see synergies with the joining the two communities and/or
> > projects?
> >
> > - Josh
> >
> > --
> > Sent from A Mobile Device
> >
> --
> Sent from A Mobile Device

Re: [DISCUSS] Is there an opportunity work together?

[Josh Fischer <jo...@joshfischer.io>]

Sending a reply to not end the email on a “yes/no” answer.

I had the not so well thought of idea of creating some sort of shared
runner.  Could we create a service or abstraction over each of the
different engines to run the same job?  What issues would we run into?
What benefits could come from this?

I’d like to hear others ideas on what could be as well.  What do each of
you think?

On Tue, Aug 30, 2022 at 12:45 PM Josh Fischer <jo...@joshfischer.io> wrote:

> Hi all,
>
> Heron is starting the process of graduating from the incubator and an idea
> was brought up by one of our mentors.
>
> Does anyone see synergies with the joining the two communities and/or
> projects?
>
> - Josh
>
> --
> Sent from A Mobile Device
>
-- 
Sent from A Mobile Device

Re: [DISCUSS] Is there an opportunity work together?

[Josh Fischer <jo...@joshfischer.io>]

Sending a reply to not end the email on a “yes/no” answer.

I had the not so well thought of idea of creating some sort of shared
runner.  Could we create a service or abstraction over each of the
different engines to run the same job?  What issues would we run into?
What benefits could come from this?

I’d like to hear others ideas on what could be as well.  What do each of
you think?

On Tue, Aug 30, 2022 at 12:45 PM Josh Fischer <jo...@joshfischer.io> wrote:

> Hi all,
>
> Heron is starting the process of graduating from the incubator and an idea
> was brought up by one of our mentors.
>
> Does anyone see synergies with the joining the two communities and/or
> projects?
>
> - Josh
>
> --
> Sent from A Mobile Device
>
-- 
Sent from A Mobile Device