You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by John Lilley <jo...@redpointglobal.com.INVALID> on 2022/11/11 15:00:38 UTC
Multi-threaded consumers in AMQ classic vs Artemis
Greetings,
We are using AMQ/Artemis to build a set of RPC-style services. We discovered that Artemis (by default) uses a global thread pool for all consumers, whereas AMQ classic created a new thread every time we make a consumer and call setMessageListener(). At least I think that’s what happened in AMQ classic. Generally I like the thread-pool model better, but it makes me question whether our current approach is still correct.
Our approach is as follows. A service processing a request queue that wants to use N threads, then for each thread we
* Get the singleton Connection
* Create a new Session
* Create a new MessageConsumer on the queue
* Create a MessageProducer to return RPC responses on the reply-to queues
* Call setMessageListener() on the consumer
Is this the best pattern for Artemis? In AMQ classic we were able to share the Session across consumers, but this is does not seem to be allowed in Artemis.
If the consumers are stateless, is there a way to get N multi-threading without creating N consumers? Or must I have a consumer per thread?
Thanks
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Multi-threaded consumers in AMQ classic vs Artemis
Posted by Clebert Suconic <cl...@gmail.com>.
TL;DR (Too long didn't read or short version): Artemis is using virtual
executors out of a pool shared on the client side. don't worry about it.
Longer version:
We implemented a virtual executor at OrderedExecutor:
https://github.com/apache/activemq-artemis/blob/main/artemis-commons/src/main/java/org/apache/activemq/artemis/utils/actors/OrderedExecutor.java
So, if you have a single Threaded executor as your pool:
if You had this virtual eecutor as in:
Executor singleThreadedPool = ...
OrderedExecutor virtual1 = new OrderedExecutor(singleThreadPool);
OrderedExecutor virtual2 = new OrderedExecutor(singleThreadPool);
And if you did:
virtual1.execute(() -> System.out.println("eecute1");
virtual1.execute(() -> System.out.println("eecute2");
virtual2.execute(() -> System.out.println("eecute1");
virtual2.execute(() -> System.out.println("eecute2");
All these runnables would finish using a single thread.
In artemis the pool has a bigger size, and you shoulddn't run out of
threades unless you are doing something crazy such as sleeping forever in
your MessageListeners. (Which I urge you to not do it).
In case you are using such anti-pattern, then you have no choice other than
increase your pool size, but you wouldn't scale as much as each thread is
using a lot of memory from your heap.
if you keep your MessageListeners just regular short tasks you don't need
to worry about it.
Increasing the pool size won't help you much anyways as you wouldn't be
able to execute more than 1 thread per Core anyways. We provide a pool a
little bigger as you sometimes have to block in certain operations such as
IO... etc... so we dimensioned the pool a little bigger than the cores
available. But creating more pools or raising your number of threads is not
going to help you.
On Sat, Nov 12, 2022 at 11:58 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Clebert,
>
>
>
> Thanks for the quick response!
>
>
>
> Let me make sure I understand (all of the code I’m talking about is using
> the JMS client API). The client thread pool consists of worker threads
> which are enlisted to call my JMS MessageListener.onMessage(). So the
> maximum number of requests that can be simultaneously serviced by all
> MessageListeners in the entire JVM is set by the thread pool limit, which
> in turn can be changed by the URI parameter scheduledThreadPoolMaxSize.
>
>
>
> But, if I understand correctly, the parallelism of given service will also
> be constrained by the number of MessageListener instances I’ve created
> and activated. So a given service’s parallelism will be limited by the
> smaller of those two numbers.
>
>
>
> This has become a concern for us given the following scenario:
>
> - We have about 25 logical RPC services combined into a single JVM
> process.
> - Each RPC service runs between 1 and 10 MessageListener instances to
> reflect desired scalability.
> - However, because we are following a blocking RPC-style pattern, some
> of these RPC services can call out to other RPC services, and block their
> handler threads while awaiting a response.
> - Under high load patterns, all available threads can be blocked
> waiting for responses that cannot be serviced due to lack of available
> threads, resulting in deadlock and eventual request timeouts.
>
>
>
> Initially our response was to increase the scheduledThreadPoolMaxSize to
> a larger number. But this just pushes the problem off to a higher load
> threshold.
>
>
>
> My current proposal is, having done a static caller/callee analysis of our
> services, to split the services into groups, each using a separate thread
> pool, so that no given group can call itself. Fortunately our call graph
> is acyclic. I believe this can be implemented, according to
> https://activemq.apache.org/components/artemis/documentation/latest/thread-pooling.html,
> by
>
> - Set useGlobalPools=false in the URI
> - Use a different ActiveMQConnectionFactory for each service group
> - Set scheduledThreadPoolMaxSize in the URI to a value sufficient to
> handle the max desired parallelism across all services
>
>
>
> Does this seem like a workable strategy? Is there a better approach?
>
>
>
> Yes, I know that async programming models solve this nicely, but we are
> not yet ready to go there for reasons of existing code base, skills, and
> experience.
>
>
>
> Thanks
>
> John
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Clebert Suconic <cl...@gmail.com>
> *Sent:* Friday, November 11, 2022 8:19 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Multi-threaded consumers in AMQ classic vs Artemis
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> The thread pool on the client is just for executors and other shared
> threads.
>
>
>
> Like if you have a MessageListener, the client will call an
> executor.execute(....); when a listener is called.
>
> So if you have multiple connections on your client (from different
> connection factories) we wouldn't be creating threads like crazy.
>
>
>
> Unless you are doing something crazy (many, many threads) this shouldn't
> be an issue. It was meant to share the thread pool between multiple clients
> and clients towards different servers.
>
>
>
>
>
> On Fri, Nov 11, 2022 at 10:01 AM John Lilley <
> john.lilley@redpointglobal.com.invalid> wrote:
>
> Greetings,
>
>
>
> We are using AMQ/Artemis to build a set of RPC-style services. We
> discovered that Artemis (by default) uses a global thread pool for all
> consumers, whereas AMQ classic created a new thread every time we make a
> consumer and call setMessageListener(). At least I *think* that’s what
> happened in AMQ classic. Generally I like the thread-pool model better,
> but it makes me question whether our current approach is still correct.
>
>
>
> Our approach is as follows. A service processing a request queue that
> wants to use N threads, then for each thread we
>
> - Get the singleton Connection
> - Create a new Session
> - Create a new MessageConsumer on the queue
> - Create a MessageProducer to return RPC responses on the reply-to
> queues
> - Call setMessageListener() on the consumer
>
>
>
> Is this the best pattern for Artemis? In AMQ classic we were able to
> share the Session across consumers, but this is does not seem to be allowed
> in Artemis.
>
>
>
> If the consumers are stateless, is there a way to get N multi-threading
> without creating N consumers? Or must I have a consumer per thread?
>
>
>
> Thanks
>
> john
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,e0TlXYG8A-_otWi0FY31Xvr786gBgUaFsYRw-dSy-eue1-Licy_JWRKBB9dlS0_ZVV1QkDAuSFfxiFnrowiWWJPWcfvtN4LzXz8ocAWrDPA9L5Na&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
>
>
> --
>
> Clebert Suconic
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
--
Clebert Suconic
RE: Multi-threaded consumers in AMQ classic vs Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Clebert,
Now that I’ve written all of this… it also occurs to me that simply setting scheduledThreadPoolMaxSize to be at least the total number of MessageListener instances would also avoid the deadlock issue, since the number of desired threads could never exceed the number of available threads. Would you agree with that, or have I missed something?
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: Saturday, November 12, 2022 9:58 AM
To: users@activemq.apache.org
Subject: RE: Multi-threaded consumers in AMQ classic vs Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Clebert,
Thanks for the quick response!
Let me make sure I understand (all of the code I’m talking about is using the JMS client API). The client thread pool consists of worker threads which are enlisted to call my JMS MessageListener.onMessage(). So the maximum number of requests that can be simultaneously serviced by all MessageListeners in the entire JVM is set by the thread pool limit, which in turn can be changed by the URI parameter scheduledThreadPoolMaxSize.
But, if I understand correctly, the parallelism of given service will also be constrained by the number of MessageListener instances I’ve created and activated. So a given service’s parallelism will be limited by the smaller of those two numbers.
This has become a concern for us given the following scenario:
* We have about 25 logical RPC services combined into a single JVM process.
* Each RPC service runs between 1 and 10 MessageListener instances to reflect desired scalability.
* However, because we are following a blocking RPC-style pattern, some of these RPC services can call out to other RPC services, and block their handler threads while awaiting a response.
* Under high load patterns, all available threads can be blocked waiting for responses that cannot be serviced due to lack of available threads, resulting in deadlock and eventual request timeouts.
Initially our response was to increase the scheduledThreadPoolMaxSize to a larger number. But this just pushes the problem off to a higher load threshold.
My current proposal is, having done a static caller/callee analysis of our services, to split the services into groups, each using a separate thread pool, so that no given group can call itself. Fortunately our call graph is acyclic. I believe this can be implemented, according to https://activemq.apache.org/components/artemis/documentation/latest/thread-pooling.html, by
* Set useGlobalPools=false in the URI
* Use a different ActiveMQConnectionFactory for each service group
* Set scheduledThreadPoolMaxSize in the URI to a value sufficient to handle the max desired parallelism across all services
Does this seem like a workable strategy? Is there a better approach?
Yes, I know that async programming models solve this nicely, but we are not yet ready to go there for reasons of existing code base, skills, and experience.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,Zottewp3WwT0B6-9n-M8CoWl3lDpOpZ6hpXlnMC1So4kUKIMWypb_5l1d6W0d1FvCOr40z9Nfgkxu-GbU1gpegNc2iJx8FzvXcq63tI_ynBvcikmrI4o&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Clebert Suconic <cl...@gmail.com>>
Sent: Friday, November 11, 2022 8:19 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Multi-threaded consumers in AMQ classic vs Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
The thread pool on the client is just for executors and other shared threads.
Like if you have a MessageListener, the client will call an executor.execute(....); when a listener is called.
So if you have multiple connections on your client (from different connection factories) we wouldn't be creating threads like crazy.
Unless you are doing something crazy (many, many threads) this shouldn't be an issue. It was meant to share the thread pool between multiple clients and clients towards different servers.
On Fri, Nov 11, 2022 at 10:01 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We are using AMQ/Artemis to build a set of RPC-style services. We discovered that Artemis (by default) uses a global thread pool for all consumers, whereas AMQ classic created a new thread every time we make a consumer and call setMessageListener(). At least I think that’s what happened in AMQ classic. Generally I like the thread-pool model better, but it makes me question whether our current approach is still correct.
Our approach is as follows. A service processing a request queue that wants to use N threads, then for each thread we
* Get the singleton Connection
* Create a new Session
* Create a new MessageConsumer on the queue
* Create a MessageProducer to return RPC responses on the reply-to queues
* Call setMessageListener() on the consumer
Is this the best pattern for Artemis? In AMQ classic we were able to share the Session across consumers, but this is does not seem to be allowed in Artemis.
If the consumers are stateless, is there a way to get N multi-threading without creating N consumers? Or must I have a consumer per thread?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,e0TlXYG8A-_otWi0FY31Xvr786gBgUaFsYRw-dSy-eue1-Licy_JWRKBB9dlS0_ZVV1QkDAuSFfxiFnrowiWWJPWcfvtN4LzXz8ocAWrDPA9L5Na&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
--
Clebert Suconic
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Multi-threaded consumers in AMQ classic vs Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Clebert,
Thanks for the quick response!
Let me make sure I understand (all of the code I’m talking about is using the JMS client API). The client thread pool consists of worker threads which are enlisted to call my JMS MessageListener.onMessage(). So the maximum number of requests that can be simultaneously serviced by all MessageListeners in the entire JVM is set by the thread pool limit, which in turn can be changed by the URI parameter scheduledThreadPoolMaxSize.
But, if I understand correctly, the parallelism of given service will also be constrained by the number of MessageListener instances I’ve created and activated. So a given service’s parallelism will be limited by the smaller of those two numbers.
This has become a concern for us given the following scenario:
* We have about 25 logical RPC services combined into a single JVM process.
* Each RPC service runs between 1 and 10 MessageListener instances to reflect desired scalability.
* However, because we are following a blocking RPC-style pattern, some of these RPC services can call out to other RPC services, and block their handler threads while awaiting a response.
* Under high load patterns, all available threads can be blocked waiting for responses that cannot be serviced due to lack of available threads, resulting in deadlock and eventual request timeouts.
Initially our response was to increase the scheduledThreadPoolMaxSize to a larger number. But this just pushes the problem off to a higher load threshold.
My current proposal is, having done a static caller/callee analysis of our services, to split the services into groups, each using a separate thread pool, so that no given group can call itself. Fortunately our call graph is acyclic. I believe this can be implemented, according to https://activemq.apache.org/components/artemis/documentation/latest/thread-pooling.html, by
* Set useGlobalPools=false in the URI
* Use a different ActiveMQConnectionFactory for each service group
* Set scheduledThreadPoolMaxSize in the URI to a value sufficient to handle the max desired parallelism across all services
Does this seem like a workable strategy? Is there a better approach?
Yes, I know that async programming models solve this nicely, but we are not yet ready to go there for reasons of existing code base, skills, and experience.
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Clebert Suconic <cl...@gmail.com>
Sent: Friday, November 11, 2022 8:19 AM
To: users@activemq.apache.org
Subject: Re: Multi-threaded consumers in AMQ classic vs Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
The thread pool on the client is just for executors and other shared threads.
Like if you have a MessageListener, the client will call an executor.execute(....); when a listener is called.
So if you have multiple connections on your client (from different connection factories) we wouldn't be creating threads like crazy.
Unless you are doing something crazy (many, many threads) this shouldn't be an issue. It was meant to share the thread pool between multiple clients and clients towards different servers.
On Fri, Nov 11, 2022 at 10:01 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We are using AMQ/Artemis to build a set of RPC-style services. We discovered that Artemis (by default) uses a global thread pool for all consumers, whereas AMQ classic created a new thread every time we make a consumer and call setMessageListener(). At least I think that’s what happened in AMQ classic. Generally I like the thread-pool model better, but it makes me question whether our current approach is still correct.
Our approach is as follows. A service processing a request queue that wants to use N threads, then for each thread we
* Get the singleton Connection
* Create a new Session
* Create a new MessageConsumer on the queue
* Create a MessageProducer to return RPC responses on the reply-to queues
* Call setMessageListener() on the consumer
Is this the best pattern for Artemis? In AMQ classic we were able to share the Session across consumers, but this is does not seem to be allowed in Artemis.
If the consumers are stateless, is there a way to get N multi-threading without creating N consumers? Or must I have a consumer per thread?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,e0TlXYG8A-_otWi0FY31Xvr786gBgUaFsYRw-dSy-eue1-Licy_JWRKBB9dlS0_ZVV1QkDAuSFfxiFnrowiWWJPWcfvtN4LzXz8ocAWrDPA9L5Na&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
--
Clebert Suconic
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Multi-threaded consumers in AMQ classic vs Artemis
Posted by Clebert Suconic <cl...@gmail.com>.
The thread pool on the client is just for executors and other shared
threads.
Like if you have a MessageListener, the client will call an
executor.execute(....); when a listener is called.
So if you have multiple connections on your client (from different
connection factories) we wouldn't be creating threads like crazy.
Unless you are doing something crazy (many, many threads) this shouldn't be
an issue. It was meant to share the thread pool between multiple clients
and clients towards different servers.
On Fri, Nov 11, 2022 at 10:01 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Greetings,
>
>
>
> We are using AMQ/Artemis to build a set of RPC-style services. We
> discovered that Artemis (by default) uses a global thread pool for all
> consumers, whereas AMQ classic created a new thread every time we make a
> consumer and call setMessageListener(). At least I *think* that’s what
> happened in AMQ classic. Generally I like the thread-pool model better,
> but it makes me question whether our current approach is still correct.
>
>
>
> Our approach is as follows. A service processing a request queue that
> wants to use N threads, then for each thread we
>
> - Get the singleton Connection
> - Create a new Session
> - Create a new MessageConsumer on the queue
> - Create a MessageProducer to return RPC responses on the reply-to
> queues
> - Call setMessageListener() on the consumer
>
>
>
> Is this the best pattern for Artemis? In AMQ classic we were able to
> share the Session across consumers, but this is does not seem to be allowed
> in Artemis.
>
>
>
> If the consumers are stateless, is there a way to get N multi-threading
> without creating N consumers? Or must I have a consumer per thread?
>
>
>
> Thanks
>
> john
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
--
Clebert Suconic