You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by Otis Gospodnetić <ot...@gmail.com> on 2017/12/29 16:29:30 UTC
Small (memory) footprint specifics
Hi,
This is from the FAQ:
*Edgent is designed for the edge. It has a small footprint, suitable for
running on constrained devices.*
Can anyone share some specifics of that "small footprint" part in terms of
memory? How small of a footprint can Edgent have?
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
Re: Small (memory) footprint specifics
Posted by Dale LaBossiere <dm...@gmail.com>.
Sorry, I don’t have any sizing numbers to offer. Don’t know if others on the list have any experiences they’d like to share.
As you expect, which connectors are used would likely have a big influence on the size. In many cases we’re at the mercy of an underlying 3rd party connector client library (e.g., MQTT, Kafka).
I would hope that you wouldn’t find it too difficult / time consuming to construct a trivial prototype app for your environment. Then you can measure it… and hopefully report the results here :-)
Note, DevelopmentSample is a poor choice for another reason: use of the DevelopmentProvider, which supplies the Edgent Console for that app, isn’t expected to be in a deployed/production Edgent application. Stick to the DirectProvider or IotProvider depending on your needs.
There are samples for several connectors. There isn’t one for the HttpStreams connector [1] but its method doc has some sample usage code.
We don’t have a JMX connector but that shouldn’t be a big impediment. E.g., a “connector” can be as simple as a function that gets a data item every time it’s called. You may have already noticed [2] (which also points to some example code using ElasticSearch), and [3]. If you end up developing a more formal JMX connector please consider contributing it!
e.g., a prototype app something along the lines of
public static void main(String[] args) throws Exception {
DirectProvider p = new DirectProvider();
Topology top = p.newTopology();
final String url = “http://httpbin.org/put”;
TStream<JsonObject> myJmxStream = topology.poll(
() -> someMethodThatGetsTheNextDataAndReturnsItAsJson(),
1, TimeUnit.SECONDS );
TStream<JsonObject> rc = HttpStreams.putJson(myJmxStream,
HttpClients::noAuthentication, t -> url, t -> t);
// myJmxStream.sink(t -> System.out.println(“Data: “ + t));
p.submit(top);
}
[1] http://edgent.apache.org/javadoc/latest/org/apache/edgent/connectors/http/HttpStreams.html
[2] http://edgent.apache.org/docs/power-of-edgent.html <http://edgent.apache.org/docs/power-of-edgent.html>
[3] http://edgent.apache.org/recipes/recipe_writing_a_connector.html <http://edgent.apache.org/recipes/recipe_writing_a_connector.html>
Hope that helps!
— Dale
> On Dec 29, 2017, at 5:05 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Unfortunately I can't provide you with such data as I am just getting started in using Edgent in real applications. Maybe Dale might help you with that.
>
> Chris
>
> Outlook for Android<https://aka.ms/ghei36> herunterladen
>
> ________________________________
> From: Otis Gospodnetić <ot...@gmail.com>
> Sent: Friday, December 29, 2017 10:36:21 PM
> To: dev@edgent.apache.org
> Subject: Re: Small (memory) footprint specifics
>
> Hi Chris,
>
> Thanks for the quick reply!
> 3.9 MB heap (was the RSS also cca 4 MB?) is much smaller than I anticipated
> given this is running in the JVM. What's the trick? :)
>
> I assume that DevelopmentSample app is the one from
> https://github.com/apache/incubator-edgent-samples/tree/develop/topology/src/main/java/org/apache/edgent/samples/topology
> ?
>
> How much bigger might the memory footprint be if one ran a more realistic
> app, say something that:
> * get data from JMX (so RMI involved) or calls some HTTP API that returns
> JSON (so parsing needed)
> * labels/tags data (like in the
> https://github.com/apache/incubator-edgent-samples/blob/develop/topology/src/main/java/org/apache/edgent/samples/topology/StreamTags.java
> )
>
> ... but still no windowed analysis on the agent side.
>
> Are we then talking 10-20-30+ MB?
>
> Thanks,
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Fri, Dec 29, 2017 at 2:38 PM, Christofer Dutz <ch...@c-ware.de>
> wrote:
>
>> Hi Otis,
>>
>> So, I just ran the DevelopmentSample application in JProfiler and it took
>> a minimum of 3,9MB of heap … This however is neither a very representative
>> Example, but it should demonstrate that you don’t need a lot of memory to
>> run an Edgent application. I guess it also depends highly on the type of
>> operations. If you use a lot of Windowed analysis, I would expect the
>> memory requirements to be a lot higher than when using only pure stream
>> operations that don’t require keeping events in memory.
>>
>> Chris
>>
>> Am 29.12.17, 17:29 schrieb "Otis Gospodnetić" <otis.gospodnetic@gmail.com
>>> :
>>
>> Hi,
>>
>> This is from the FAQ:
>>
>> *Edgent is designed for the edge. It has a small footprint, suitable
>> for
>> running on constrained devices.*
>>
>> Can anyone share some specifics of that "small footprint" part in
>> terms of
>> memory? How small of a footprint can Edgent have?
>>
>> Thanks,
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training -
>> http://sematext.com/
>>
>>
>>
Re: Small (memory) footprint specifics
Posted by Christofer Dutz <ch...@c-ware.de>.
Unfortunately I can't provide you with such data as I am just getting started in using Edgent in real applications. Maybe Dale might help you with that.
Chris
Outlook for Android<https://aka.ms/ghei36> herunterladen
________________________________
From: Otis Gospodnetić <ot...@gmail.com>
Sent: Friday, December 29, 2017 10:36:21 PM
To: dev@edgent.apache.org
Subject: Re: Small (memory) footprint specifics
Hi Chris,
Thanks for the quick reply!
3.9 MB heap (was the RSS also cca 4 MB?) is much smaller than I anticipated
given this is running in the JVM. What's the trick? :)
I assume that DevelopmentSample app is the one from
https://github.com/apache/incubator-edgent-samples/tree/develop/topology/src/main/java/org/apache/edgent/samples/topology
?
How much bigger might the memory footprint be if one ran a more realistic
app, say something that:
* get data from JMX (so RMI involved) or calls some HTTP API that returns
JSON (so parsing needed)
* labels/tags data (like in the
https://github.com/apache/incubator-edgent-samples/blob/develop/topology/src/main/java/org/apache/edgent/samples/topology/StreamTags.java
)
... but still no windowed analysis on the agent side.
Are we then talking 10-20-30+ MB?
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Fri, Dec 29, 2017 at 2:38 PM, Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi Otis,
>
> So, I just ran the DevelopmentSample application in JProfiler and it took
> a minimum of 3,9MB of heap … This however is neither a very representative
> Example, but it should demonstrate that you don’t need a lot of memory to
> run an Edgent application. I guess it also depends highly on the type of
> operations. If you use a lot of Windowed analysis, I would expect the
> memory requirements to be a lot higher than when using only pure stream
> operations that don’t require keeping events in memory.
>
> Chris
>
> Am 29.12.17, 17:29 schrieb "Otis Gospodnetić" <otis.gospodnetic@gmail.com
> >:
>
> Hi,
>
> This is from the FAQ:
>
> *Edgent is designed for the edge. It has a small footprint, suitable
> for
> running on constrained devices.*
>
> Can anyone share some specifics of that "small footprint" part in
> terms of
> memory? How small of a footprint can Edgent have?
>
> Thanks,
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
>
>
>
Re: Small (memory) footprint specifics
Posted by Otis Gospodnetić <ot...@gmail.com>.
Hi Chris,
Thanks for the quick reply!
3.9 MB heap (was the RSS also cca 4 MB?) is much smaller than I anticipated
given this is running in the JVM. What's the trick? :)
I assume that DevelopmentSample app is the one from
https://github.com/apache/incubator-edgent-samples/tree/develop/topology/src/main/java/org/apache/edgent/samples/topology
?
How much bigger might the memory footprint be if one ran a more realistic
app, say something that:
* get data from JMX (so RMI involved) or calls some HTTP API that returns
JSON (so parsing needed)
* labels/tags data (like in the
https://github.com/apache/incubator-edgent-samples/blob/develop/topology/src/main/java/org/apache/edgent/samples/topology/StreamTags.java
)
... but still no windowed analysis on the agent side.
Are we then talking 10-20-30+ MB?
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Fri, Dec 29, 2017 at 2:38 PM, Christofer Dutz <ch...@c-ware.de>
wrote:
> Hi Otis,
>
> So, I just ran the DevelopmentSample application in JProfiler and it took
> a minimum of 3,9MB of heap … This however is neither a very representative
> Example, but it should demonstrate that you don’t need a lot of memory to
> run an Edgent application. I guess it also depends highly on the type of
> operations. If you use a lot of Windowed analysis, I would expect the
> memory requirements to be a lot higher than when using only pure stream
> operations that don’t require keeping events in memory.
>
> Chris
>
> Am 29.12.17, 17:29 schrieb "Otis Gospodnetić" <otis.gospodnetic@gmail.com
> >:
>
> Hi,
>
> This is from the FAQ:
>
> *Edgent is designed for the edge. It has a small footprint, suitable
> for
> running on constrained devices.*
>
> Can anyone share some specifics of that "small footprint" part in
> terms of
> memory? How small of a footprint can Edgent have?
>
> Thanks,
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
>
>
>
Re: Small (memory) footprint specifics
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Otis,
So, I just ran the DevelopmentSample application in JProfiler and it took a minimum of 3,9MB of heap … This however is neither a very representative Example, but it should demonstrate that you don’t need a lot of memory to run an Edgent application. I guess it also depends highly on the type of operations. If you use a lot of Windowed analysis, I would expect the memory requirements to be a lot higher than when using only pure stream operations that don’t require keeping events in memory.
Chris
Am 29.12.17, 17:29 schrieb "Otis Gospodnetić" <ot...@gmail.com>:
Hi,
This is from the FAQ:
*Edgent is designed for the edge. It has a small footprint, suitable for
running on constrained devices.*
Can anyone share some specifics of that "small footprint" part in terms of
memory? How small of a footprint can Edgent have?
Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/