You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Lahiru Ginnaliya Gamathige <gl...@gmail.com> on 2016/10/04 16:29:26 UTC
KafkaAppender for Airavata
Hi Devs,
I've been working on a feature in Airavata to send its logs to a Kafka
topic and finally move them to elastic search using logstash. Please find
all the work[1] and the code[2] and I've added a documentation in to wiki
[3](since we do not have a development location for wiki).
After finishing the issue, Me and Ajinkya did a setup with Airavata and
elastic search and Kafka cluster and we were able to visualize the logs in
kibana.
Please provide your valuable feedback on this feature.
[1]https://issues.apache.org/jira/browse/AIRAVATA-2065
[2]
https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
[3]
https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
Regards
Lahiru
Re: KafkaAppender for Airavata
Posted by Shameera Rathnayaka <sh...@gmail.com>.
On Fri, Oct 7, 2016 at 10:15 AM Lahiru Ginnaliya Gamathige <
glahiru@gmail.com> wrote:
> Created a Jira for RabbitMQ Appender[1], please implement it if you like.
>
It is always good to plug differnt implementations, If I am going to
implement a new logging appender(RabbitMQAppender or any other). How I can
plug it to Airavata? Can we easily plug differnt implementations? It seem
you have code agains the kafka logging implementation instead of an
Interface, correct me if I am wrong. I think if you can provide better
solution to plug different appenders this task would be more easier.
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2145
>
> On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>
>
> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> Hi Lahiru,
>
> I didn't see any architectural mail regarding this hence asking, Can you
> explain your design and deployment strategy here? few follow-up questions,
> how many kafka producers, partitions and consumers in the system?, why you
> choose kafka over other messaging systems?
>
> One producer in-process with Airavata JVM(if we deploy gfac, orchestrator,
> airavata-api separate then one produce per each component), consumer is in
> logstash which reads the logs from preconfigured topics. Once we push to
> Kafka its totally up the the developers who deploy airavata to use it
> however they want. Using logstash and ES is just one approach which me and
> Ajinkya tried and a very common solution people use to visualize their logs
> coming from multiple components.
>
>
> How many instances need to deploy your logging solution in productions?
> How Simple/Complex compare to Airavata deployment?
>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>
>
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs. I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
> Yes obvious statement.
>
> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
> Regards,
> Shameera.
>
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
> Shameera Rathnayaka
>
> --
> Shameera Rathnayaka
>
>
>
> --
Shameera Rathnayaka
Re: KafkaAppender for Airavata
Posted by Lahiru Ginnaliya Gamathige <gl...@gmail.com>.
Created a Jira for RabbitMQ Appender[1], please implement it if you like.
[1]https://issues.apache.org/jira/browse/AIRAVATA-2145
On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
glahiru@gmail.com> wrote:
>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>>
>>
>> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>> Hi Lahiru,
>>>
>>> I didn't see any architectural mail regarding this hence asking, Can you
>>> explain your design and deployment strategy here? few follow-up questions,
>>> how many kafka producers, partitions and consumers in the system?, why you
>>> choose kafka over other messaging systems?
>>>
>>> One producer in-process with Airavata JVM(if we deploy gfac,
>>> orchestrator, airavata-api separate then one produce per each component),
>>> consumer is in logstash which reads the logs from preconfigured topics.
>>> Once we push to Kafka its totally up the the developers who deploy airavata
>>> to use it however they want. Using logstash and ES is just one approach
>>> which me and Ajinkya tried and a very common solution people use to
>>> visualize their logs coming from multiple components.
>>>
>>
>> How many instances need to deploy your logging solution in productions?
>> How Simple/Complex compare to Airavata deployment?
>>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>>
>>
>
>>> Regarding why Kafka, I've used similar setup and it never gave me a
>>> problem and worked with thousands of logs without giving performance
>>> problems or issue with the scale of the logs. I did a demo/presentation
>>> about this to the IU folks few weeks back and attached the presentation to
>>> Jira[1].
>>>
>>
>> Just because Kafka works doesn't mean kafka is the best solution for
>> Airavata.
>>
> Yes obvious statement.
>
>> If you explain little bit the setup you have worked on, that would help
>> us to get some ideas. What is the other advantage make Kafka is the best
>> one here instead of RabbitMQ (Which we already use in Airavata for
>> messaging) ?
>>
>> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
>> Regards,
>> Shameera.
>>
>>
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>>
>>> Regards,
>>> Shameera.
>>>
>>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
>>> glahiru@gmail.com> wrote:
>>>
>>> Hi Suresh,
>>>
>>> If logstash see back-pressure then it stops processing the messages so
>>> its always good to have a persistant storage as the input side of the
>>> logstash, so logstash can crash anytime and kafka offset works very nicely
>>> with the cases where logstash crashes and you just have to restart the
>>> logstash and missed logs will be loaded to ES cluster.
>>>
>>> There's nothing like sending the logs *to* logstash because logstash is
>>> simply a data pipeline and not a data store. The other reason is that if we
>>> push to kafka we can just handle moving the data to ES with minimum number
>>> of logstash (theoretically one logstash) where we have the full
>>> configuration with what topics to listen and where to send those logs.
>>>
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>> Hi Lahiru,
>>>
>>> Can you please elaborate on the role of the messaging system (Kafka in
>>> your example) vs directly using log stash plugins [1] as an example?
>>>
>>> Suresh
>>> [1] - https://www.elastic.co/guide/en/logstash/current/plugins-
>>> inputs-log4j.html
>>>
>>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>>> glahiru@gmail.com> wrote:
>>>
>>> Hi Devs,
>>>
>>> I've been working on a feature in Airavata to send its logs to a Kafka
>>> topic and finally move them to elastic search using logstash. Please find
>>> all the work[1] and the code[2] and I've added a documentation in to wiki
>>> [3](since we do not have a development location for wiki).
>>>
>>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>>> elastic search and Kafka cluster and we were able to visualize the logs in
>>> kibana.
>>>
>>> Please provide your valuable feedback on this feature.
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073
>>> 784795dfc56618ec556a12ad1
>>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto
>>> rial+09+-+How+to+setup+centralized+logging+for+Airavata
>>>
>>>
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>> --
>>> Shameera Rathnayaka
>>>
>>> --
>> Shameera Rathnayaka
>>
>
>
Re: KafkaAppender for Airavata
Posted by Shameera Rathnayaka <sh...@gmail.com>.
How many instances need to deploy your logging solution in productions? How
> Simple/Complex compare to Airavata deployment?
>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
Do this solution need only kafka? I am interested in deployment setup for
full solution not just for kafka. As you already worked on similar steup
previously(you mentioned above) i though you might have recommended
deployment on board.
>
>
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs. I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
> Yes obvious statement.
>
Yes, you gave that obvious statement for the original quection "why
kafka?". Do you have any other reason than this obvious statement.
> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too.
>
Could you explain more about this?
> Having KafkaAppender will not stop you from implementing more Appenders,
> there's more improvements can be done in this area to provide better
> infrastructure with Airavata.
>
But we should have default solution in Airavata, that is the whole point of
this integration, anyone can write any appender. When i see you solution I
don't see an easy way to plug different appenders.
Thanks,
Shameera.
> Regards,
> Shameera.
>
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
> Shameera Rathnayaka
>
> --
> Shameera Rathnayaka
>
> --
Shameera Rathnayaka
Re: KafkaAppender for Airavata
Posted by Lahiru Ginnaliya Gamathige <gl...@gmail.com>.
On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <sh...@gmail.com>
wrote:
>
>
> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>> Hi Lahiru,
>>
>> I didn't see any architectural mail regarding this hence asking, Can you
>> explain your design and deployment strategy here? few follow-up questions,
>> how many kafka producers, partitions and consumers in the system?, why you
>> choose kafka over other messaging systems?
>>
>> One producer in-process with Airavata JVM(if we deploy gfac,
>> orchestrator, airavata-api separate then one produce per each component),
>> consumer is in logstash which reads the logs from preconfigured topics.
>> Once we push to Kafka its totally up the the developers who deploy airavata
>> to use it however they want. Using logstash and ES is just one approach
>> which me and Ajinkya tried and a very common solution people use to
>> visualize their logs coming from multiple components.
>>
>
> How many instances need to deploy your logging solution in productions?
> How Simple/Complex compare to Airavata deployment?
>
These are completely deployment level details which someone needs to
experiment with based on the load of Airavata deployment, for the setup I
tested I used 3 node kafka cluster with 25GB in each and I think its more
than enough but completely depends on the deployment.
>
>
>> Regarding why Kafka, I've used similar setup and it never gave me a
>> problem and worked with thousands of logs without giving performance
>> problems or issue with the scale of the logs. I did a demo/presentation
>> about this to the IU folks few weeks back and attached the presentation to
>> Jira[1].
>>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
Yes obvious statement.
> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
should support Kafka for internal communication too. Having KafkaAppender
will not stop you from implementing more Appenders, there's more
improvements can be done in this area to provide better infrastructure with
Airavata.
> Regards,
> Shameera.
>
>
>>
>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>> Regards
>> Lahiru
>>
>>
>>
>>
>> Regards,
>> Shameera.
>>
>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>> Hi Suresh,
>>
>> If logstash see back-pressure then it stops processing the messages so
>> its always good to have a persistant storage as the input side of the
>> logstash, so logstash can crash anytime and kafka offset works very nicely
>> with the cases where logstash crashes and you just have to restart the
>> logstash and missed logs will be loaded to ES cluster.
>>
>> There's nothing like sending the logs *to* logstash because logstash is
>> simply a data pipeline and not a data store. The other reason is that if we
>> push to kafka we can just handle moving the data to ES with minimum number
>> of logstash (theoretically one logstash) where we have the full
>> configuration with what topics to listen and where to send those logs.
>>
>> Regards
>> Lahiru
>>
>>
>>
>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>> Hi Lahiru,
>>
>> Can you please elaborate on the role of the messaging system (Kafka in
>> your example) vs directly using log stash plugins [1] as an example?
>>
>> Suresh
>> [1] - https://www.elastic.co/guide/en/logstash/current/
>> plugins-inputs-log4j.html
>>
>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>> Hi Devs,
>>
>> I've been working on a feature in Airavata to send its logs to a Kafka
>> topic and finally move them to elastic search using logstash. Please find
>> all the work[1] and the code[2] and I've added a documentation in to wiki
>> [3](since we do not have a development location for wiki).
>>
>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>> elastic search and Kafka cluster and we were able to visualize the logs in
>> kibana.
>>
>> Please provide your valuable feedback on this feature.
>>
>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>> [2]https://github.com/apache/airavata/commit/
>> 8eea17fb9c8c073784795dfc56618ec556a12ad1
>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/
>> Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>>
>>
>> Regards
>> Lahiru
>>
>>
>>
>> --
>> Shameera Rathnayaka
>>
>> --
> Shameera Rathnayaka
>
Re: KafkaAppender for Airavata
Posted by Shameera Rathnayaka <sh...@gmail.com>.
On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <gl...@gmail.com>
wrote:
> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> Hi Lahiru,
>
> I didn't see any architectural mail regarding this hence asking, Can you
> explain your design and deployment strategy here? few follow-up questions,
> how many kafka producers, partitions and consumers in the system?, why you
> choose kafka over other messaging systems?
>
> One producer in-process with Airavata JVM(if we deploy gfac, orchestrator,
> airavata-api separate then one produce per each component), consumer is in
> logstash which reads the logs from preconfigured topics. Once we push to
> Kafka its totally up the the developers who deploy airavata to use it
> however they want. Using logstash and ES is just one approach which me and
> Ajinkya tried and a very common solution people use to visualize their logs
> coming from multiple components.
>
How many instances need to deploy your logging solution in productions? How
Simple/Complex compare to Airavata deployment?
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs. I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
Just because Kafka works doesn't mean kafka is the best solution for
Airavata. If you explain little bit the setup you have worked on, that
would help us to get some ideas. What is the other advantage make Kafka is
the best one here instead of RabbitMQ (Which we already use in Airavata for
messaging) ?
Regards,
Shameera.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
> Shameera Rathnayaka
>
> --
Shameera Rathnayaka
Re: KafkaAppender for Airavata
Posted by Lahiru Ginnaliya Gamathige <gl...@gmail.com>.
On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Lahiru,
>
> I didn't see any architectural mail regarding this hence asking, Can you
> explain your design and deployment strategy here? few follow-up questions,
> how many kafka producers, partitions and consumers in the system?, why you
> choose kafka over other messaging systems?
>
One producer in-process with Airavata JVM(if we deploy gfac, orchestrator,
airavata-api separate then one produce per each component), consumer is in
logstash which reads the logs from preconfigured topics. Once we push to
Kafka its totally up the the developers who deploy airavata to use it
however they want. Using logstash and ES is just one approach which me and
Ajinkya tried and a very common solution people use to visualize their logs
coming from multiple components.
Regarding why Kafka, I've used similar setup and it never gave me a problem
and worked with thousands of logs without giving performance problems or
issue with the scale of the logs. I did a demo/presentation about this to
the IU folks few weeks back and attached the presentation to Jira[1].
[1]https://issues.apache.org/jira/browse/AIRAVATA-2065
Regards
Lahiru
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
>> Hi Suresh,
>>
>> If logstash see back-pressure then it stops processing the messages so
>> its always good to have a persistant storage as the input side of the
>> logstash, so logstash can crash anytime and kafka offset works very nicely
>> with the cases where logstash crashes and you just have to restart the
>> logstash and missed logs will be loaded to ES cluster.
>>
>> There's nothing like sending the logs *to* logstash because logstash is
>> simply a data pipeline and not a data store. The other reason is that if we
>> push to kafka we can just handle moving the data to ES with minimum number
>> of logstash (theoretically one logstash) where we have the full
>> configuration with what topics to listen and where to send those logs.
>>
>> Regards
>> Lahiru
>>
>>
>>
>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>> Hi Lahiru,
>>
>> Can you please elaborate on the role of the messaging system (Kafka in
>> your example) vs directly using log stash plugins [1] as an example?
>>
>> Suresh
>> [1] - https://www.elastic.co/guide/en/logstash/current/plugins-
>> inputs-log4j.html
>>
>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>> Hi Devs,
>>
>> I've been working on a feature in Airavata to send its logs to a Kafka
>> topic and finally move them to elastic search using logstash. Please find
>> all the work[1] and the code[2] and I've added a documentation in to wiki
>> [3](since we do not have a development location for wiki).
>>
>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>> elastic search and Kafka cluster and we were able to visualize the logs in
>> kibana.
>>
>> Please provide your valuable feedback on this feature.
>>
>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073
>> 784795dfc56618ec556a12ad1
>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto
>> rial+09+-+How+to+setup+centralized+logging+for+Airavata
>>
>>
>> Regards
>> Lahiru
>>
>>
>>
>> --
> Shameera Rathnayaka
>
Re: KafkaAppender for Airavata
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Lahiru,
I didn't see any architectural mail regarding this hence asking, Can you
explain your design and deployment strategy here? few follow-up questions,
how many kafka producers, partitions and consumers in the system?, why you
choose kafka over other messaging systems?
Regards,
Shameera.
On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <gl...@gmail.com>
wrote:
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
Shameera Rathnayaka
Re: KafkaAppender for Airavata
Posted by Shameera Rathnayaka <sh...@gmail.com>.
On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <gl...@gmail.com>
wrote:
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
This behavior is common to all available AMQP tools. With Rabbitmq writing
the similar behavior is easier than kafka, where this redelivery is handled
by Rabbitmq instead of consumers.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
Shameera Rathnayaka
Re: KafkaAppender for Airavata
Posted by Lahiru Ginnaliya Gamathige <gl...@gmail.com>.
Hi Suresh,
If logstash see back-pressure then it stops processing the messages so its
always good to have a persistant storage as the input side of the logstash,
so logstash can crash anytime and kafka offset works very nicely with the
cases where logstash crashes and you just have to restart the logstash and
missed logs will be loaded to ES cluster.
There's nothing like sending the logs *to* logstash because logstash is
simply a data pipeline and not a data store. The other reason is that if we
push to kafka we can just handle moving the data to ES with minimum number
of logstash (theoretically one logstash) where we have the full
configuration with what topics to listen and where to send those logs.
Regards
Lahiru
On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] - https://www.elastic.co/guide/en/logstash/current/
> plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]https://github.com/apache/airavata/commit/
> 8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/
> Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
Re: KafkaAppender for Airavata
Posted by Suresh Marru <sm...@apache.org>.
Hi Lahiru,
Can you please elaborate on the role of the messaging system (Kafka in your example) vs directly using log stash plugins [1] as an example?
Suresh
[1] - https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html <https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <gl...@gmail.com> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka topic and finally move them to elastic search using logstash. Please find all the work[1] and the code[2] and I've added a documentation in to wiki [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and elastic search and Kafka cluster and we were able to visualize the logs in kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 <https://issues.apache.org/jira/browse/AIRAVATA-2065>
> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1 <https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1>
> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata <https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata>
>
>
> Regards
> Lahiru