You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@heron.apache.org by Josh Fischer <jo...@joshfischer.io> on 2021/07/21 19:10:11 UTC
Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc
for us all to discuss. If you still have interest in adding the Heron
Simulator to ECO please review this doc and make comments as needed. It's
not meant to be "this is how it's going to be done", it's more of a
conversation starter to get us all on the same page of what needs to be
done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com> wrote:
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Please share what you find, what your thoughts are, etc. I haven't used
the simulator much, so I'd definitely be interested in learning more about
it and how you use it, plan to use it and anything else you run into.
On Mon, Aug 9, 2021 at 8:40 AM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh, I'm tentatively interested in this - would be a good
> learning experience for sure.
>
> I'll need to dig around in the Storm code a bit to see the complexity here.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 5, 2021 1:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> How would you feel about implementing the functionality into Heron and
> submitting a PR back to the project? I can help along the way as needed.
>
> On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh, I've reviewed and looks fine to me.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 2, 2021 4:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> Ning left feedback. You want to check it out for me?
>
> On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Please share what you find, what your thoughts are, etc. I haven't used
the simulator much, so I'd definitely be interested in learning more about
it and how you use it, plan to use it and anything else you run into.
On Mon, Aug 9, 2021 at 8:40 AM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh, I'm tentatively interested in this - would be a good
> learning experience for sure.
>
> I'll need to dig around in the Storm code a bit to see the complexity here.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 5, 2021 1:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> How would you feel about implementing the functionality into Heron and
> submitting a PR back to the project? I can help along the way as needed.
>
> On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh, I've reviewed and looks fine to me.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 2, 2021 4:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> Ning left feedback. You want to check it out for me?
>
> On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh, I'm tentatively interested in this - would be a good learning experience for sure.
I'll need to dig around in the Storm code a bit to see the complexity here.
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: August 5, 2021 1:25 PM
To: Scott Mayne <ro...@hotmail.com>
Cc: user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
How would you feel about implementing the functionality into Heron and submitting a PR back to the project? I can help along the way as needed.
On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh, I've reviewed and looks fine to me.
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: August 2, 2021 4:25 PM
To: Scott Mayne <ro...@hotmail.com>>
Cc: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; romulan_ace@hotmail.com<ma...@hotmail.com> <ro...@hotmail.com>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh, I'm tentatively interested in this - would be a good learning experience for sure.
I'll need to dig around in the Storm code a bit to see the complexity here.
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: August 5, 2021 1:25 PM
To: Scott Mayne <ro...@hotmail.com>
Cc: user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
How would you feel about implementing the functionality into Heron and submitting a PR back to the project? I can help along the way as needed.
On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh, I've reviewed and looks fine to me.
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: August 2, 2021 4:25 PM
To: Scott Mayne <ro...@hotmail.com>>
Cc: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; romulan_ace@hotmail.com<ma...@hotmail.com> <ro...@hotmail.com>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
How would you feel about implementing the functionality into Heron and
submitting a PR back to the project? I can help along the way as needed.
On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh, I've reviewed and looks fine to me.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 2, 2021 4:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> Ning left feedback. You want to check it out for me?
>
> On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
How would you feel about implementing the functionality into Heron and
submitting a PR back to the project? I can help along the way as needed.
On Tue, Aug 3, 2021 at 2:29 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh, I've reviewed and looks fine to me.
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* August 2, 2021 4:25 PM
> *To:* Scott Mayne <ro...@hotmail.com>
> *Cc:* user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org
> <de...@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> Ning left feedback. You want to check it out for me?
>
> On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh, I've reviewed and looks fine to me.
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: August 2, 2021 4:25 PM
To: Scott Mayne <ro...@hotmail.com>
Cc: user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; romulan_ace@hotmail.com<ma...@hotmail.com> <ro...@hotmail.com>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh, I've reviewed and looks fine to me.
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: August 2, 2021 4:25 PM
To: Scott Mayne <ro...@hotmail.com>
Cc: user@heron.apache.org <us...@heron.apache.org>; dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; romulan_ace@hotmail.com<ma...@hotmail.com> <ro...@hotmail.com>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Ning, Nick, Huijun, Windham, Oliver,
Could any of you take a look at this if you have time? Or anyone else if
they feel so inclined. I have a few other questions I think will need to be
addressed, but I’ll bring them up down the road.
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
> --
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Scott,
Ning left feedback. You want to check it out for me?
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
>
Re: ECO + YAML in IDE
Posted by Josh Fischer <jo...@joshfischer.io>.
Ning, Nick, Huijun, Windham, Oliver,
Could any of you take a look at this if you have time? Or anyone else if
they feel so inclined. I have a few other questions I think will need to be
addressed, but I’ll bring them up down the road.
On Sun, Jul 25, 2021 at 4:47 PM Scott Mayne <ro...@hotmail.com> wrote:
> Hi Josh,
>
> The doc looks good to me (others can weigh in on the few open questions, I
> have no strong feelings one way or another).
>
> Definitely still interested in having ECO YAML + Simulator functionality.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* July 21, 2021 3:10 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>;
> romulan_ace@hotmail.com <ro...@hotmail.com>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Scott,
>
> I want to apologize, I took entirely too long to come up with a design doc
> for us all to discuss. If you still have interest in adding the Heron
> Simulator to ECO please review this doc and make comments as needed. It's
> not meant to be "this is how it's going to be done", it's more of a
> conversation starter to get us all on the same page of what needs to be
> done.
>
>
> https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
>
> If anyone else has any questions, comments, or concerns, please let me
> know.
>
> - Josh
>
> On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Thanks Josh, sounds good.
>
> Apologies for my over-describing, just wanted to make sure I was being
> clear.
>
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 8, 2021 2:57 PM
> *To:* user@heron.apache.org <us...@heron.apache.org>
> *Cc:* dev@heron.apache.org <de...@heron.apache.org>;
> user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Got it. My thinking out loud via email doesn't always work out so well.
> I'll come up with a design doc to share that we can all talk about and
> hopefully clarify things. Give me a few days and I'll get it out to the
> list.
>
> Hope that works for you,
>
> - Josh
>
> On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi Josh,
>
> My first reaction is for it to not be specified in the topology YAML
> (seems easier to handle if nothing changes in that file, however deployed).
>
> But it's certainly not a dealbreaker and will defer to your opinion as to
> what's best from an overall Heron point of view.
>
> This new Simulator functionality in EcoSubmitter, it will allow an
> environment-specific HeronConfig file to be passed in?
>
> And the Flux syntax didn't need a JAR path with the "local" invocation,
> just to reiterate the goal here is to have Eco use the IDE workspace files
> (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
>
> Thanks,
> Scott
>
> ------------------------------
> *From:* Josh Fischer <jo...@joshfischer.io>
> *Sent:* June 6, 2021 10:10 PM
> *To:* dev@heron.apache.org <de...@heron.apache.org>; user@heron.apache.org <
> user@heron.apache.org>; user@heron.incubator.apache.org <
> user@heron.incubator.apache.org>
> *Subject:* Re: ECO + YAML in IDE
>
> Hi Scott,
>
> I think we would need to make changes to the EcoSubmitter class to allow
> for use of the Simulator. How would you prefer to specify that topology
> would be run in the simulator? Would you prefer to specify this in the
> topology yaml file or would you rather specify it as a command line
> argument, something like
> """
> heron submit local \
> some/path/to/jar.jar \
> org.apache.heron.eco.Eco \
> --eco path/to/topology.yaml --simulate
> """
> If we pass --simulate as a parameter for the Eco deployment I have a
> concern that it could be confusing for the other deployment methods since
> it would be ignored for something like the Java Topology API as we would
> specify this in the java code, not a deployment parameter.
>
>
> - Josh
>
>
>
> On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
> Hi Scott,
>
>
> I’ll have time to look in to this tomorrow for you. I’ve also sent this to
> the dev@heron list in case someone else would be interested.
>
> - Josh
>
>
> On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>
> wrote:
>
> Hi all,
>
> I asked a question about running ECO in local mode a while back, thanks to
> Josh for suggesting the verbose flag but nothing was added to the console
> that shed any light on why it exited right away.
>
> In the meantime, I got things running in IntelliJ
> using org.apache.heron.simulator.Simulator, after creating the topology
> using org.apache.heron.api.topology.TopologyBuilder.
>
> Now that that's working, we've circled back to the ECO/IDE/YAML question.
> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for
> cluster deploys, and in an IDE with a main() method using Flux as well (see
> bottom of message for my unsuccessful attempt to mimic the Flux way with
> Eco).
>
> Obviously Storm 1.0.6 is getting a little long in the tooth and we're
> exploring moving to Heron (and liking what we see so far).
>
> A nice thing about the Storm setup was we use one YAML file to define the
> topologies in both scenarios. So far in Heron, it would be ECO/YAML for
> cluster (not that we've gotten that far yet) and TopologyBuilder in the
> IDE for development.
>
> Would be nice to not have this difference and use YAML for both (avoid
> mismatches between the two, etc.), so my question is: Has anyone set up
> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology
> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If
> so, how?
>
> As background, these Storm ETLs were written before I joined the team, and
> I'm relatively new to streaming frameworks, so I might be missing something
> obvious here.
>
> For example, I just removed the Storm artifacts from the POM, added Heron,
> and modified the ETL code as needed. Haven't installed Heron binaries or
> anything like that on my laptop, not sure if that's a prerequisite.
>
> Thanks,
> Scott
>
> public class KafkaTopology {
>
> public static void main(String[] args) throws Exception {
>
> List<String> argList = new ArrayList<String>(Arrays.asList(args));
> String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
> argList.add("local");
> argList.add("--eco-config-file");
> argList.add(file);
>
> file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
> argList.add("--props");
> argList.add(file);
>
> argList.add("--sleep"); // something from Flux to make it run for a while
> argList.add("36000000");
>
> String[] ecoArgs = argList.toArray(new String[argList.size()]);
>
> Eco.main(ecoArgs);
>
> }
> }
>
>
> --
> Sent from A Mobile Device
>
> --
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org <us...@heron.apache.org>; romulan_ace@hotmail.com <ro...@hotmail.com>
Cc: dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device
Re: ECO + YAML in IDE
Posted by Scott Mayne <ro...@hotmail.com>.
Hi Josh,
The doc looks good to me (others can weigh in on the few open questions, I have no strong feelings one way or another).
Definitely still interested in having ECO YAML + Simulator functionality.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>
Sent: July 21, 2021 3:10 PM
To: user@heron.apache.org <us...@heron.apache.org>; romulan_ace@hotmail.com <ro...@hotmail.com>
Cc: dev@heron.apache.org <de...@heron.apache.org>; user@heron.incubator.apache.org <us...@heron.incubator.apache.org>
Subject: Re: ECO + YAML in IDE
Scott,
I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing
If anyone else has any questions, comments, or concerns, please let me know.
- Josh
On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <ro...@hotmail.com>> wrote:
Thanks Josh, sounds good.
Apologies for my over-describing, just wanted to make sure I was being clear.
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 8, 2021 2:57 PM
To: user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>
Cc: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you,
- Josh
On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <ro...@hotmail.com>> wrote:
Hi Josh,
My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view.
This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in?
And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc.
Thanks,
Scott
________________________________
From: Josh Fischer <jo...@joshfischer.io>>
Sent: June 6, 2021 10:10 PM
To: dev@heron.apache.org<ma...@heron.apache.org> <de...@heron.apache.org>>; user@heron.apache.org<ma...@heron.apache.org> <us...@heron.apache.org>>; user@heron.incubator.apache.org<ma...@heron.incubator.apache.org> <us...@heron.incubator.apache.org>>
Subject: Re: ECO + YAML in IDE
Hi Scott,
I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like
"""
heron submit local \
some/path/to/jar.jar \
org.apache.heron.eco.Eco \
--eco path/to/topology.yaml --simulate
"""
If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh
On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <jo...@joshfischer.io>> wrote:
Hi Scott,
I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested.
- Josh
On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <ro...@hotmail.com>> wrote:
Hi all,
I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away.
In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder.
Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco).
Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far).
A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development.
Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how?
As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here.
For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite.
Thanks,
Scott
public class KafkaTopology {
public static void main(String[] args) throws Exception {
List<String> argList = new ArrayList<String>(Arrays.asList(args));
String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile();
argList.add("local");
argList.add("--eco-config-file");
argList.add(file);
file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile();
argList.add("--props");
argList.add(file);
argList.add("--sleep"); // something from Flux to make it run for a while
argList.add("36000000");
String[] ecoArgs = argList.toArray(new String[argList.size()]);
Eco.main(ecoArgs);
}
}
--
Sent from A Mobile Device