You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by Julian Feinauer <j....@pragmaticminds.de> on 2018/08/06 11:49:51 UTC

USP Apache Edgent

Hi all,

this is my first post on the Apache Edgent mailing list and first, let me say, that I’m quite impressed by all that has been done with regards to IoT and Edge Processing in this project.
As I observe the project for a long time now and we start to see many use cases with our customers we decided to get deeper in touch with the community.

When reasoning about the architecture of a new system for our new costumers including a lot of “IoT” communication we obviously came by apache edgent.
Can you say, what in your opinion the unique selling point of edgent is?
Is the focus on the interaction with Cloud Based IoT Services or do you see the low footprint stream processing also as one advantage of edgent?
I tried to get a really good overview over the remote control capabilities and the MXBean integration, but I think I’m not at it, yet.

We looked at different stream processing platforms like Flink, Spark (which are everything except low footprint) but also Akka streams which seems to fit very well with “low footprint”.
So do you see Akka streams as an alternative to Edgent when one only uses a “Simple” Topology with a DirectProvider and no integration to Remote Services or Control and Command Servers or anything?

I would love to hear your opinion on that!

Best
Julian

pragmatic minds GmbH

Re: USP Apache Edgent

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,



I would really like to give you input about the internals and best practices of Edgent, however I can't :-(



I started working on Edgent after hearing from it in Miami and had been concentrating on fixing the build system (Did the Ant->Maven conversion). So I really understand 100% of the build system, but shortly after releasing 1.2.0 the others seem to have shifted their priorities. So right now deep knowledge of Edgent is scarce on this list. I am trying to dig in a little deeper every day, but as you know ... most of my time currently goes into PLC4X. But for me Edgent is THE tool for writing PLC4X Applications and I'm not going to abandon it.



Interesting that you mention a drop-in replacement of Kafka, as I was planning on suggesting this on this list. As for me there is a direct correlation between Kafka Streams and Edgent Streams, I was planning on suggesting to streamline the naming of things as well as the syntax. So someone able to write applications for one of the two will be able to achieve the same in the other.



Chris







Am 07.08.18, 08:39 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:



    Hey Chris,

    

    

    

    hello to you on this list and thanks for your response.

    

    I get your point and I agree that Akka streams is very complex which is especially bad for simple use cases.

    

    But, as you also say, Edgent and Akka streams seem to be the first options for streaming in situations where one has limited resources (which is also often times the situation for us).

    

    

    

    What I liked about Edgent when I first looked into it (must be 2 years ago, I think shortly after it was renamed from Quartz) I liked the command & control approach but up till now I'm not able to understand the concept (it instruments something wich MBeans and JMX it seems but I can't figure it out). Do you have any hints for "good" documentation on that or at least some code samples which help to get the principle?

    

    

    

    Another point I am currently missing in the IoT Landscape is a (logical) drop in replacement for something like Apache Kafka. We use it usually for two reasons. First, the obvious decoupling from data generation and processing (and the possibility to backpressure) and second as a message store (in the concept of Event Sourcing) to allow for later reprocessing.

    

    Stripping it down, without all the distributed stuff Kafka is "just" an ordered queue (as Jay Kreps likes to emphasize also). So in the context of stream processing it would love to have a Sink / Source with semantics similar to kafkas (producer, consumer, topics, offset management) which is backed by a simple append only file. One interesting Implementation of such a queue (without the exact semantics, I think, is e.g., [1]).

    

    Is there something similar in Edgent currently, I did not observe anything like that.

    

    

    

    Would you think that this is something which would fit in Edgents scope as IoT Streaming project?

    

    

    

    Best

    

    Julian

    

    

    

    [1] https://github.com/OpenHFT/Chronicle-Queue

    

    

    

    Am 06.08.18, 23:32 schrieb "Christofer Dutz" <ch...@c-ware.de>:

    

    

    

        Hi Julian,

    

        

    

        

    

        

    

        welcome here on the Edgent list :-)

    

        

    

        

    

        

    

        Well I can't speak for the rest of the team, but for me Edgent is currently sort of the following:

    

        

    

        - For me Edgent is the ideal entry point for any sort of IoT system 

    

        

    

        - I like to compare it with Kakfa-Connect, but without the cloud (ok we can read and write to clouds, but Edgent is not a cloud based system)

    

        

    

        

    

        

    

        Especially if you want to operate in constrained environments, such as: 

    

        

    

        - Limited processing power

    

        

    

        - Low energy consumption

    

        

    

        - Low memory consumption

    

        

    

        - Narrow/Unreliable/High-Latency network connectivity

    

        

    

        Then is where you can cross all the cloud based streaming frameworks off your list.

    

        

    

        

    

        

    

        Regarding Akka Streams ... well the presentations I did some time ago with my little Rapiro Robot, were based on Akka Streams ... that was why I fell in love with Edgent when I heard about it at ApacheCon NA in Miami. For me, Akka Streams totally fries my neurons. It's far too complicated. And as soon as you want to work with Fan-In/Fan-Out and Backpressure ... well ... I thought it was nuts ... but I have to admit that I'm no Akka or Scala pro, so please don't treat this assumption as a fact. It's just my opinion. 

    

        

    

        

    

        

    

        Does that answer your question? If not ... feel free to ask :)

    

        

    

        

    

        

    

        Chris

    

        

    

        

    

        

    

        

    

        

    

        Am 06.08.18, 13:50 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:

    

        

    

        

    

        

    

            Hi all,

    

        

    

            

    

        

    

            this is my first post on the Apache Edgent mailing list and first, let me say, that I’m quite impressed by all that has been done with regards to IoT and Edge Processing in this project.

    

        

    

            As I observe the project for a long time now and we start to see many use cases with our customers we decided to get deeper in touch with the community.

    

        

    

            

    

        

    

            When reasoning about the architecture of a new system for our new costumers including a lot of “IoT” communication we obviously came by apache edgent.

    

        

    

            Can you say, what in your opinion the unique selling point of edgent is?

    

        

    

            Is the focus on the interaction with Cloud Based IoT Services or do you see the low footprint stream processing also as one advantage of edgent?

    

        

    

            I tried to get a really good overview over the remote control capabilities and the MXBean integration, but I think I’m not at it, yet.

    

        

    

            

    

        

    

            We looked at different stream processing platforms like Flink, Spark (which are everything except low footprint) but also Akka streams which seems to fit very well with “low footprint”.

    

        

    

            So do you see Akka streams as an alternative to Edgent when one only uses a “Simple” Topology with a DirectProvider and no integration to Remote Services or Control and Command Servers or anything?

    

        

    

            

    

        

    

            I would love to hear your opinion on that!

    

        

    

            

    

        

    

            Best

    

        

    

            Julian

    

        

    

            

    

        

    

            pragmatic minds GmbH

Re: USP Apache Edgent

Posted by Julian Feinauer <j....@pragmaticminds.de>.
Hey Chris,



hello to you on this list and thanks for your response.

I get your point and I agree that Akka streams is very complex which is especially bad for simple use cases.

But, as you also say, Edgent and Akka streams seem to be the first options for streaming in situations where one has limited resources (which is also often times the situation for us).



What I liked about Edgent when I first looked into it (must be 2 years ago, I think shortly after it was renamed from Quartz) I liked the command & control approach but up till now I'm not able to understand the concept (it instruments something wich MBeans and JMX it seems but I can't figure it out). Do you have any hints for "good" documentation on that or at least some code samples which help to get the principle?



Another point I am currently missing in the IoT Landscape is a (logical) drop in replacement for something like Apache Kafka. We use it usually for two reasons. First, the obvious decoupling from data generation and processing (and the possibility to backpressure) and second as a message store (in the concept of Event Sourcing) to allow for later reprocessing.

Stripping it down, without all the distributed stuff Kafka is "just" an ordered queue (as Jay Kreps likes to emphasize also). So in the context of stream processing it would love to have a Sink / Source with semantics similar to kafkas (producer, consumer, topics, offset management) which is backed by a simple append only file. One interesting Implementation of such a queue (without the exact semantics, I think, is e.g., [1]).

Is there something similar in Edgent currently, I did not observe anything like that.



Would you think that this is something which would fit in Edgents scope as IoT Streaming project?



Best

Julian



[1] https://github.com/OpenHFT/Chronicle-Queue



Am 06.08.18, 23:32 schrieb "Christofer Dutz" <ch...@c-ware.de>:



    Hi Julian,

    

    

    

    welcome here on the Edgent list :-)

    

    

    

    Well I can't speak for the rest of the team, but for me Edgent is currently sort of the following:

    

    - For me Edgent is the ideal entry point for any sort of IoT system 

    

    - I like to compare it with Kakfa-Connect, but without the cloud (ok we can read and write to clouds, but Edgent is not a cloud based system)

    

    

    

    Especially if you want to operate in constrained environments, such as: 

    

    - Limited processing power

    

    - Low energy consumption

    

    - Low memory consumption

    

    - Narrow/Unreliable/High-Latency network connectivity

    

    Then is where you can cross all the cloud based streaming frameworks off your list.

    

    

    

    Regarding Akka Streams ... well the presentations I did some time ago with my little Rapiro Robot, were based on Akka Streams ... that was why I fell in love with Edgent when I heard about it at ApacheCon NA in Miami. For me, Akka Streams totally fries my neurons. It's far too complicated. And as soon as you want to work with Fan-In/Fan-Out and Backpressure ... well ... I thought it was nuts ... but I have to admit that I'm no Akka or Scala pro, so please don't treat this assumption as a fact. It's just my opinion. 

    

    

    

    Does that answer your question? If not ... feel free to ask :)

    

    

    

    Chris

    

    

    

    

    

    Am 06.08.18, 13:50 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:

    

    

    

        Hi all,

    

        

    

        this is my first post on the Apache Edgent mailing list and first, let me say, that I’m quite impressed by all that has been done with regards to IoT and Edge Processing in this project.

    

        As I observe the project for a long time now and we start to see many use cases with our customers we decided to get deeper in touch with the community.

    

        

    

        When reasoning about the architecture of a new system for our new costumers including a lot of “IoT” communication we obviously came by apache edgent.

    

        Can you say, what in your opinion the unique selling point of edgent is?

    

        Is the focus on the interaction with Cloud Based IoT Services or do you see the low footprint stream processing also as one advantage of edgent?

    

        I tried to get a really good overview over the remote control capabilities and the MXBean integration, but I think I’m not at it, yet.

    

        

    

        We looked at different stream processing platforms like Flink, Spark (which are everything except low footprint) but also Akka streams which seems to fit very well with “low footprint”.

    

        So do you see Akka streams as an alternative to Edgent when one only uses a “Simple” Topology with a DirectProvider and no integration to Remote Services or Control and Command Servers or anything?

    

        

    

        I would love to hear your opinion on that!

    

        

    

        Best

    

        Julian

    

        

    

        pragmatic minds GmbH

Re: USP Apache Edgent

Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Julian,



welcome here on the Edgent list :-)



Well I can't speak for the rest of the team, but for me Edgent is currently sort of the following:

- For me Edgent is the ideal entry point for any sort of IoT system 

- I like to compare it with Kakfa-Connect, but without the cloud (ok we can read and write to clouds, but Edgent is not a cloud based system)



Especially if you want to operate in constrained environments, such as: 

- Limited processing power

- Low energy consumption

- Low memory consumption

- Narrow/Unreliable/High-Latency network connectivity

Then is where you can cross all the cloud based streaming frameworks off your list.



Regarding Akka Streams ... well the presentations I did some time ago with my little Rapiro Robot, were based on Akka Streams ... that was why I fell in love with Edgent when I heard about it at ApacheCon NA in Miami. For me, Akka Streams totally fries my neurons. It's far too complicated. And as soon as you want to work with Fan-In/Fan-Out and Backpressure ... well ... I thought it was nuts ... but I have to admit that I'm no Akka or Scala pro, so please don't treat this assumption as a fact. It's just my opinion. 



Does that answer your question? If not ... feel free to ask :)



Chris





Am 06.08.18, 13:50 schrieb "Julian Feinauer" <j....@pragmaticminds.de>:



    Hi all,

    

    this is my first post on the Apache Edgent mailing list and first, let me say, that I’m quite impressed by all that has been done with regards to IoT and Edge Processing in this project.

    As I observe the project for a long time now and we start to see many use cases with our customers we decided to get deeper in touch with the community.

    

    When reasoning about the architecture of a new system for our new costumers including a lot of “IoT” communication we obviously came by apache edgent.

    Can you say, what in your opinion the unique selling point of edgent is?

    Is the focus on the interaction with Cloud Based IoT Services or do you see the low footprint stream processing also as one advantage of edgent?

    I tried to get a really good overview over the remote control capabilities and the MXBean integration, but I think I’m not at it, yet.

    

    We looked at different stream processing platforms like Flink, Spark (which are everything except low footprint) but also Akka streams which seems to fit very well with “low footprint”.

    So do you see Akka streams as an alternative to Edgent when one only uses a “Simple” Topology with a DirectProvider and no integration to Remote Services or Control and Command Servers or anything?

    

    I would love to hear your opinion on that!

    

    Best

    Julian

    

    pragmatic minds GmbH