You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Gustavo Beneitez <gu...@gmail.com> on 2018/09/04 07:50:28 UTC
Kafka and logstash integration
Hello Karl,
is there any kind of integration in the roadmap between manifoldCF and
Kafka or LogStash technologies?
We are trying to expand replications with several technologies in mind.
Thanks!
Re: Kafka and logstash integration
Posted by Gustavo Beneitez <gu...@gmail.com>.
Hi Piergiorgio,
thanks for your response. I also agree Kafka is best for this solution.
In fact, we are facing Elastic cluster problems and developers told me it
might be possible to persist / distribute messages through kafka instead of
directly dealing with Elastic. Also, someone pointed to a logstash solution
as ELK does, but you are right, Kafka should be the way.
As I understand, logstash should "replace" or "complement" web crawler by
just putting "events" into the system that might be transformed into
documents in parallel with other repositories. I like a lot a passive
connector, some kind of listener that could get individual document
requests and incorporate them to the system, not necessary to be tied with
Logstash.
Best regards!
El vie., 7 sept. 2018 a las 14:25, Piergiorgio Lucidi (<
piergiorgio@apache.org>) escribió:
> Hi Gustavo,
>
> nice to meet you!
>
> We actually have a Kafka Repository Connector as mentioned in our latest
> compatibility matrix:
>
> https://manifoldcf.apache.org/release/release-2.10/en_US/included-connectors.html
>
> Considering Logstash we don't have nothing at the moment but I can imagine
> the scenario behind it.
> The architecture for this case is quite different because ManifoldCF is
> fundamentally a repository crawler and the interaction with logstash should
> be inline from logstash to ManifoldCF that is not actually supported for
> each single transaction.
>
> We could think about a passive connector that remains listening to receive
> new contents, anyway we should start talking about which could the best
> approach for implementing this new behavior.
>
> Am I missing something?
> Hope to receive more comments from other folks in the community :)
>
> Cheers,
> PJ
>
>
>
>
> Il giorno mar 4 set 2018 alle ore 09:50 Gustavo Beneitez <
> gustavo.beneitez@gmail.com> ha scritto:
>
> > Hello Karl,
> >
> > is there any kind of integration in the roadmap between manifoldCF and
> > Kafka or LogStash technologies?
> >
> > We are trying to expand replications with several technologies in mind.
> >
> > Thanks!
> >
>
>
> --
> Piergiorgio Lucidi
> Open Source Evangelist and Digital Transformation Specialist
> Member / Mentor / PMC Member / Committer @ The Apache Software Foundation
> Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco
> Author and Technical Reviewer @ Packt Publishing
> Technical Advisory Group Member @ Microsoft
> Top Community Contributor @ Crafter
> Project Leader / Committer @ JBoss
> https://www.open4dev.com
>
Re: Kafka and logstash integration
Posted by Piergiorgio Lucidi <pi...@apache.org>.
Hi Gustavo,
nice to meet you!
We actually have a Kafka Repository Connector as mentioned in our latest
compatibility matrix:
https://manifoldcf.apache.org/release/release-2.10/en_US/included-connectors.html
Considering Logstash we don't have nothing at the moment but I can imagine
the scenario behind it.
The architecture for this case is quite different because ManifoldCF is
fundamentally a repository crawler and the interaction with logstash should
be inline from logstash to ManifoldCF that is not actually supported for
each single transaction.
We could think about a passive connector that remains listening to receive
new contents, anyway we should start talking about which could the best
approach for implementing this new behavior.
Am I missing something?
Hope to receive more comments from other folks in the community :)
Cheers,
PJ
Il giorno mar 4 set 2018 alle ore 09:50 Gustavo Beneitez <
gustavo.beneitez@gmail.com> ha scritto:
> Hello Karl,
>
> is there any kind of integration in the roadmap between manifoldCF and
> Kafka or LogStash technologies?
>
> We are trying to expand replications with several technologies in mind.
>
> Thanks!
>
--
Piergiorgio Lucidi
Open Source Evangelist and Digital Transformation Specialist
Member / Mentor / PMC Member / Committer @ The Apache Software Foundation
Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco
Author and Technical Reviewer @ Packt Publishing
Technical Advisory Group Member @ Microsoft
Top Community Contributor @ Crafter
Project Leader / Committer @ JBoss
https://www.open4dev.com