You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Norman Maurer <no...@apache.org> on 2010/02/18 16:38:05 UTC
[PROPOSAL] Using Camel as replacement of MailProcessor / SpoolManager
Hi all,
in the past Stefano told me about his Idea of using camel within james
(it was just an idea and he had no closer look yet). The last days I
did some research about camel (camel.apache.org) and I think he is
right.
So me plan is:
* Write a Consumer for our SpoolRepository which would fire of an
exchange object when a mail object is ready for processing
* Write a RouteBuilder which will create a route for every configured
processor and so replace the MailProcessor stuff at all. The route
would just to the Matcher, Mailet processing
With using Camel it would be possible (just an example) to load
balance between routes, throttle processing of low priority routes,
process mail on other server etc. I think this would push James
extensibility to a new level, and will allow developers to customize
james more easy.
WDYTH ?
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [PROPOSAL] Using Camel as replacement of MailProcessor /
SpoolManager
Posted by Norman Maurer <no...@googlemail.com>.
Hi Stefano,
comments inside...
2010/2/19 Stefano Bagnara <ap...@bago.org>:
> 2010/2/18 Norman Maurer <no...@apache.org>:
>> Hi all,
>>
>> in the past Stefano told me about his Idea of using camel within james
>> (it was just an idea and he had no closer look yet). The last days I
>> did some research about camel (camel.apache.org) and I think he is
>> right.
>
> To be true the idea was not mine, but from James Strachan:
> http://markmail.org/message/vzzmwm3kdv66r3ce
> (expecially last senteces).
Thx for the pointer to the archives..
>
>> So me plan is:
>>
>> * Write a Consumer for our SpoolRepository which would fire of an
>> exchange object when a mail object is ready for processing
>> * Write a RouteBuilder which will create a route for every configured
>> processor and so replace the MailProcessor stuff at all. The route
>> would just to the Matcher, Mailet processing
>
> IIUC you are trying to keep using the current repository
> implementation but use camel for routing. This sounds like an
> intermediate steps, not sure it's easier (as I never had the time to
> investigate on Camel internals). IIUC in your plan you keep using our
> spoolmanager xml definition to describe how processors are and then
> use camel as an alternative implementation of the "LinearProcessor"
> (and you would still use the StateAwareProcessorList and our
> repositories).
Not quite right. I would remove the LinearProcessor and
StateAwareProcessorList at all. Let me outline the idea a bit more:
I would just parse the spoolmanager.xml and create routes per
processor via the RouteBuilder class. So it would be one Route per
processor. Which processor to use will be detect by check the Camel
message header.
So I would write some kind of Camel Processor Mailet wrapper to call
the mailet, and a Camel Splitter to split the Mail for the Matcher
stuff.
They Mail objects will get "injected" in the Camel route by an
SpoolRepositoryConsumer. The Consumer will fire the Mail to the Route
when the spoolRepository.accept() return a Mail object.
>
> The original idea was to simply replace the whole spooling and
> SpoolManager with a Camel based component. This would mean dropping
> the SpoolRepository interface at all and just use Camel Transport
> (activemq/JMS as we probably want persistent queues).
I agree this would be the "final" solution. I just want todo the
refactoring step by step to get a better overview and idea about it.
Replace the SpoolRepositoryConsumer with something else should be
really easy once the rest is in place..
>
> MailServer and MailetContext services then would change to send the
> message to Camel or maybe we should change the 2 services with a
> simpler service send(Mail/Envelope/Exchange message,
> String/URL/Address destination).
+1
>
> As a POC this could work but to be production ready we'll have to also
> make the envelope/message split and (like Robert say in his reply)
> stream the message content to some storage and only route around the
> envelope (the main issue here is reference counting when we duplicate
> the envelope as this happens a lot in james).
>
True enough.. But I think the KISS principal is the way to go for
now.. We can refactor it later to make it more performant.
> Take this only as "brainstorming", if you already have a clear idea on
> the plan go ahead with your plan: we need some real experience with
> camel to see if it really is as good as it sounds :-)
>
> Stefano
>
> ---------------------------------------------------------------------
Thx for you comments..
Bye,
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [PROPOSAL] Using Camel as replacement of MailProcessor /
SpoolManager
Posted by Stefano Bagnara <ap...@bago.org>.
2010/2/18 Norman Maurer <no...@apache.org>:
> Hi all,
>
> in the past Stefano told me about his Idea of using camel within james
> (it was just an idea and he had no closer look yet). The last days I
> did some research about camel (camel.apache.org) and I think he is
> right.
To be true the idea was not mine, but from James Strachan:
http://markmail.org/message/vzzmwm3kdv66r3ce
(expecially last senteces).
> So me plan is:
>
> * Write a Consumer for our SpoolRepository which would fire of an
> exchange object when a mail object is ready for processing
> * Write a RouteBuilder which will create a route for every configured
> processor and so replace the MailProcessor stuff at all. The route
> would just to the Matcher, Mailet processing
IIUC you are trying to keep using the current repository
implementation but use camel for routing. This sounds like an
intermediate steps, not sure it's easier (as I never had the time to
investigate on Camel internals). IIUC in your plan you keep using our
spoolmanager xml definition to describe how processors are and then
use camel as an alternative implementation of the "LinearProcessor"
(and you would still use the StateAwareProcessorList and our
repositories).
The original idea was to simply replace the whole spooling and
SpoolManager with a Camel based component. This would mean dropping
the SpoolRepository interface at all and just use Camel Transport
(activemq/JMS as we probably want persistent queues).
MailServer and MailetContext services then would change to send the
message to Camel or maybe we should change the 2 services with a
simpler service send(Mail/Envelope/Exchange message,
String/URL/Address destination).
As a POC this could work but to be production ready we'll have to also
make the envelope/message split and (like Robert say in his reply)
stream the message content to some storage and only route around the
envelope (the main issue here is reference counting when we duplicate
the envelope as this happens a lot in james).
Take this only as "brainstorming", if you already have a clear idea on
the plan go ahead with your plan: we need some real experience with
camel to see if it really is as good as it sounds :-)
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: AW: [PROPOSAL] Using Camel as replacement of MailProcessor / SpoolManager
Posted by David Jencks <da...@yahoo.com>.
I'm not a camel expert.
On Feb 18, 2010, at 12:22 PM, Mario Zsilak wrote:
> Hi,
>
> I just took a glance at camel.apache.org but I spotted the following
> statement:
>
> <!-- start -->
> Apache Camel can be used as a routing and mediation engine for the
> following
> projects:
>
> * ...
> * Apache ActiveMQ which is the most popular and powerful open
> source
> message broker
> * ...
> <!-- end -->
>
>
> I guess that means that other message brokers (including commercial
> ones)
> are not supported within Apache Camel?
I think it means that the camel integration is shipped with those
projects and is not a comment on the ease of using camel with other
message systems. For instance I doubt tibco is going to ship camel
any time soon, but I have the impression that integrating with a jms
provider is basically a matter of configuration.
> In that case I would be happy to see some kind of abstraction layer
> so that
> others (like me) can implement their own stuff ...
Where would this abstraction layer go, and what would it abstract?
What would you substitute for what?
david jencks
>
> Apart from my requirements I guess most people don't need these
> (very nice)
> features at all.
> However if there is a performance gain, especially when using only 1
> server/james-instance, we should go for it.
>
> Kind regards,
> Mario
>
>> -----Ursprüngliche Nachricht-----
>> Von: norman.maurer@googlemail.com [mailto:norman.maurer@googlemail.com
>> ]
>> Im Auftrag von Norman Maurer
>> Gesendet: Donnerstag, 18. Februar 2010 16:38
>> An: James Developers List
>> Betreff: [PROPOSAL] Using Camel as replacement of MailProcessor /
>> SpoolManager
>>
>> Hi all,
>>
>> in the past Stefano told me about his Idea of using camel within
>> james
>> (it was just an idea and he had no closer look yet). The last days I
>> did some research about camel (camel.apache.org) and I think he is
>> right.
>>
>> So me plan is:
>>
>> * Write a Consumer for our SpoolRepository which would fire of an
>> exchange object when a mail object is ready for processing
>> * Write a RouteBuilder which will create a route for every configured
>> processor and so replace the MailProcessor stuff at all. The route
>> would just to the Matcher, Mailet processing
>>
>> With using Camel it would be possible (just an example) to load
>> balance between routes, throttle processing of low priority routes,
>> process mail on other server etc. I think this would push James
>> extensibility to a new level, and will allow developers to customize
>> james more easy.
>>
>> WDYTH ?
>>
>> Norman
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
AW: [PROPOSAL] Using Camel as replacement of MailProcessor / SpoolManager
Posted by Mario Zsilak <ma...@inew-cs.com>.
Hi,
I just took a glance at camel.apache.org but I spotted the following
statement:
<!-- start -->
Apache Camel can be used as a routing and mediation engine for the following
projects:
* ...
* Apache ActiveMQ which is the most popular and powerful open source
message broker
* ...
<!-- end -->
I guess that means that other message brokers (including commercial ones)
are not supported within Apache Camel?
In that case I would be happy to see some kind of abstraction layer so that
others (like me) can implement their own stuff ...
Apart from my requirements I guess most people don't need these (very nice)
features at all.
However if there is a performance gain, especially when using only 1
server/james-instance, we should go for it.
Kind regards,
Mario
>-----Ursprüngliche Nachricht-----
>Von: norman.maurer@googlemail.com [mailto:norman.maurer@googlemail.com]
>Im Auftrag von Norman Maurer
>Gesendet: Donnerstag, 18. Februar 2010 16:38
>An: James Developers List
>Betreff: [PROPOSAL] Using Camel as replacement of MailProcessor /
>SpoolManager
>
>Hi all,
>
>in the past Stefano told me about his Idea of using camel within james
>(it was just an idea and he had no closer look yet). The last days I
>did some research about camel (camel.apache.org) and I think he is
>right.
>
>So me plan is:
>
>* Write a Consumer for our SpoolRepository which would fire of an
>exchange object when a mail object is ready for processing
>* Write a RouteBuilder which will create a route for every configured
>processor and so replace the MailProcessor stuff at all. The route
>would just to the Matcher, Mailet processing
>
>With using Camel it would be possible (just an example) to load
>balance between routes, throttle processing of low priority routes,
>process mail on other server etc. I think this would push James
>extensibility to a new level, and will allow developers to customize
>james more easy.
>
>WDYTH ?
>
>Norman
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>For additional commands, e-mail: server-dev-help@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Bounce mail...problem...
Posted by Norman Maurer <no...@googlemail.com>.
Hi Robert,
what version of james ? Can you show us the bounce ? Can you show us
your config.xml ?
Without more infos its a bit hard to tell.
Bye,
Norman
Ps: server-user at apache.org is the "right" ml for such kind of
questions. server-dev is more for develop related questions
2010/2/23 Robert L. Gurule <rg...@lanl.gov>:
> I'm having a problem when I send mail to my test user the server gives me
> a jamesserversconfigsetting.com through bounces....
>
> when I look at the dnsserver log I have: Any ideas?
>
> Robert
>
>
> INFO 16:51:14,140 | james.dnsserver | DNS Server is: 192.168.0.1
> INFO 16:51:14,140 | james.dnsserver | DNS Server is: 128.165.4.4
> INFO 16:51:14,140 | james.dnsserver | DNS Server is: 128.165.11.88
> INFO 16:51:14,140 | james.dnsserver | Registered cache, resolver and
> search paths as DNSJava defaults
> INFO 16:53:31,031 | james.dnsserver | Couldn't resolve MX records for
> domain syncrhonicity.lanl.gov.
> INFO 16:54:25,125 | james.dnsserver | Couldn't resolve MX records for
> domain JAMESServersConfigSetting.com.
> ERROR 16:54:25,125 | james.dnsserver | Couldn't resolve IP address for
> host JAMESServersConfigSetting.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Bounce mail...problem...
Posted by "Robert L. Gurule" <rg...@lanl.gov>.
I'm having a problem when I send mail to my test user the server gives me
a jamesserversconfigsetting.com through bounces....
when I look at the dnsserver log I have: Any ideas?
Robert
INFO 16:51:14,140 | james.dnsserver | DNS Server is: 192.168.0.1
INFO 16:51:14,140 | james.dnsserver | DNS Server is: 128.165.4.4
INFO 16:51:14,140 | james.dnsserver | DNS Server is: 128.165.11.88
INFO 16:51:14,140 | james.dnsserver | Registered cache, resolver and
search paths as DNSJava defaults
INFO 16:53:31,031 | james.dnsserver | Couldn't resolve MX records for
domain syncrhonicity.lanl.gov.
INFO 16:54:25,125 | james.dnsserver | Couldn't resolve MX records for
domain JAMESServersConfigSetting.com.
ERROR 16:54:25,125 | james.dnsserver | Couldn't resolve IP address for
host JAMESServersConfigSetting.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [PROPOSAL] Using Camel as replacement of MailProcessor /
SpoolManager
Posted by Norman Maurer <no...@googlemail.com>.
The POC is now in trunk. I'm using in production since today and so
far I don't see any problems.
I think I will now start to tackle the jms stuff and remove "old" code..
Bye,
Norman
2010/2/19 Norman Maurer <no...@googlemail.com>:
> Hi Robert,
>
> I 100% agree with you. But I think thats something we could take care
> of once the camel routing stuff is in place..
>
> Bye,
> Norman
>
> 2010/2/18 Robert Burrell Donkin <ro...@gmail.com>:
>> On Thu, Feb 18, 2010 at 8:22 PM, Mario Zsilak <ma...@inew-cs.com> wrote:
>>> Hi,
>>>
>>> I just took a glance at camel.apache.org but I spotted the following
>>> statement:
>>>
>>> <!-- start -->
>>> Apache Camel can be used as a routing and mediation engine for the following
>>> projects:
>>>
>>> * ...
>>> * Apache ActiveMQ which is the most popular and powerful open source
>>> message broker
>>> * ...
>>> <!-- end -->
>>>
>>>
>>> I guess that means that other message brokers (including commercial ones)
>>> are not supported within Apache Camel?
>>> In that case I would be happy to see some kind of abstraction layer so that
>>> others (like me) can implement their own stuff ...
>>>
>>> Apart from my requirements I guess most people don't need these (very nice)
>>> features at all.
>>> However if there is a performance gain, especially when using only 1
>>> server/james-instance, we should go for it.
>>
>> IIRC we've discuss this before and the consensus was that an API would
>> be the right way to go
>>
>> +1 with a few comments
>>
>> might need to think a little about big emails. might be better just to
>> send the meta-data around keeping a reference to a data store. this
>> would fit in with ideas about streaming bodies more directly into
>> storage.
>>
>> - robert
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [PROPOSAL] Using Camel as replacement of MailProcessor /
SpoolManager
Posted by Norman Maurer <no...@googlemail.com>.
Hi Robert,
I 100% agree with you. But I think thats something we could take care
of once the camel routing stuff is in place..
Bye,
Norman
2010/2/18 Robert Burrell Donkin <ro...@gmail.com>:
> On Thu, Feb 18, 2010 at 8:22 PM, Mario Zsilak <ma...@inew-cs.com> wrote:
>> Hi,
>>
>> I just took a glance at camel.apache.org but I spotted the following
>> statement:
>>
>> <!-- start -->
>> Apache Camel can be used as a routing and mediation engine for the following
>> projects:
>>
>> * ...
>> * Apache ActiveMQ which is the most popular and powerful open source
>> message broker
>> * ...
>> <!-- end -->
>>
>>
>> I guess that means that other message brokers (including commercial ones)
>> are not supported within Apache Camel?
>> In that case I would be happy to see some kind of abstraction layer so that
>> others (like me) can implement their own stuff ...
>>
>> Apart from my requirements I guess most people don't need these (very nice)
>> features at all.
>> However if there is a performance gain, especially when using only 1
>> server/james-instance, we should go for it.
>
> IIRC we've discuss this before and the consensus was that an API would
> be the right way to go
>
> +1 with a few comments
>
> might need to think a little about big emails. might be better just to
> send the meta-data around keeping a reference to a data store. this
> would fit in with ideas about streaming bodies more directly into
> storage.
>
> - robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [PROPOSAL] Using Camel as replacement of MailProcessor /
SpoolManager
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Thu, Feb 18, 2010 at 8:22 PM, Mario Zsilak <ma...@inew-cs.com> wrote:
> Hi,
>
> I just took a glance at camel.apache.org but I spotted the following
> statement:
>
> <!-- start -->
> Apache Camel can be used as a routing and mediation engine for the following
> projects:
>
> * ...
> * Apache ActiveMQ which is the most popular and powerful open source
> message broker
> * ...
> <!-- end -->
>
>
> I guess that means that other message brokers (including commercial ones)
> are not supported within Apache Camel?
> In that case I would be happy to see some kind of abstraction layer so that
> others (like me) can implement their own stuff ...
>
> Apart from my requirements I guess most people don't need these (very nice)
> features at all.
> However if there is a performance gain, especially when using only 1
> server/james-instance, we should go for it.
IIRC we've discuss this before and the consensus was that an API would
be the right way to go
+1 with a few comments
might need to think a little about big emails. might be better just to
send the meta-data around keeping a reference to a data store. this
would fit in with ideas about streaming bodies more directly into
storage.
- robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org