You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Serge Perepel <se...@american-data.com> on 2018/01/26 15:11:37 UTC

Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

When we run our app for 2-3 hours we experience this leak related to our web sockets connections, here is the Memory Analyzer Suspect:

One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>".

Keywords
org.apache.coyote.AbstractProtocol$ConnectionHandler
java.util.concurrent.ConcurrentHashMap$Node[]
java.net.URLClassLoader @ 0x720029098


Also these bugs might be related to this one:



https://bz.apache.org/bugzilla/show_bug.cgi?id=57546

https://bz.apache.org/bugzilla/show_bug.cgi?id=57750



The 57546 bug looks very similar to what we are experiencing. We tested on Linux and so far we do not see the same behavior. Also this is our web socket code:



package ad.ecs.async.websocket;



import java.io.IOException;

import java.util.Arrays;



import javax.websocket.CloseReason;

import javax.websocket.OnClose;

import javax.websocket.OnError;

import javax.websocket.OnMessage;

import javax.websocket.OnOpen;

import javax.websocket.Session;

import javax.websocket.server.ServerEndpoint;



import ad.common.Global;

import ad.ecs.async.AsyncEngine;

import ad.ecs.async.AsyncResponse;

import ad.ecs.async.AsyncType;

import ad.ecs.db.DatabaseEngine;

import ad.ecs.db.paradox.User;

import ad.ecs.security.engine.SecurityEngine;



/**

 * @author Serge Perepel

 * @since Aug 23, 2017 12:23:32 PM

 */

@ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)

public class ECSAsync {



    @OnOpen

    public void open(Session session) throws IOException{

        session.getBasicRemote().sendText("Connection Established");

    }



    @OnMessage

    public String login(String sessionID, Session session) {

        AsyncEngine.INSTANCE.wsConnect(session, sessionID);

        org.hibernate.Session dbSession = DatabaseEngine.getSessionFactory().openSession();

        try {

               int userID = SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);

               User user = (User) dbSession.get(User.class, userID);

               if (user != null) {

                       if (user.getNextLogin() == 1) {

                               AsyncResponse response = new AsyncResponse();

                               response.setType(AsyncType.None);

                               response.setData(Arrays.asList("PASSWORD"));

                               response.setObjData("PASSWORD");

                               AsyncEngine.INSTANCE.addTransientResult(sessionID, response);

                       }

               }

        } finally {

               dbSession.close();

        }

        return "ok";

    }



    @OnClose

    public void close(Session session, CloseReason reason) {

        AsyncEngine.INSTANCE.wsDisconnect(session);

    }



    @OnError

    public void error(Session session, Throwable error) {

        Global.INSTANCE.getLogHelper().exception(error);

        //session.close(new CloseReason(closeCode, reasonPhrase));

    }

}



Front end opens connection and sends invalid sessionID which causes

the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw exception and after that disconnect happens on the front end. At this point front end opens new connection and process goes into the loop. I'm assuming that the connection handler suppose to get freed after a disconnect happen. But it seems to accumulate. You can try this code and just replace code in the login method to always throw the exception. On the front end onDisconnect try to open new connection and send a random message to the web socket.



Thank you

Serge

[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>

Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
serge@american-data.com<ma...@american-data.com> | www.american-data.com


[Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>


Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are not
the intended recipient, you should not use, disclose or take any action based on
the message. If you have received this transmission in error, please immediately
contact the sender by return e-mail and delete this e-mail,
and any attachments, from any computer.

[ECS 10 Conversion]<http://www.american-data.com/viewer>

RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

Posted by Serge Perepel <se...@american-data.com>.
-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]


>A break down of that ~2Gb would be helpful. How many entries in the Map?

>What is the distribution of sizes?



Here is the screenshot, let me know if it's not what you want:

[cid:image001.png@01D3969B.943E3B40]





>This is likely to get looked at faster if you provide code that other people can run that will reproduce the issue you are seeing rather than expecting someone else to construct the test case for you.



We are going to make small test app that replicates the issue. Ill add it as soon as we finish it.





[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>

Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
serge@american-data.com<ma...@american-data.com> | www.american-data.com


[Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>


Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are not
the intended recipient, you should not use, disclose or take any action based on
the message. If you have received this transmission in error, please immediately
contact the sender by return e-mail and delete this e-mail,
and any attachments, from any computer.

[ECS 10 Conversion]<http://www.american-data.com/viewer>

RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

Posted by Serge Perepel <se...@american-data.com>.
Any takers to tackle this issue?

-----Original Message-----
From: Serge Perepel [mailto:serge@american-data.com] 
Sent: Friday, January 26, 2018 2:33 PM
To: Tomcat Users List
Subject: RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]

>This is likely to get looked at faster if you provide code that other people can run that will reproduce the issue you are seeing rather than expecting someone else to construct the test case for you.

Here is how you can reproduce it:

Server websocket:

package ad.ecs.websocket;

import java.io.IOException;

import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;

@ServerEndpoint(value = "/asyncMsg")
public class TestWebsocket {
    @OnOpen
    public void open(Session session) throws IOException{
        session.getBasicRemote().sendText("Connection Established");
    }

    @OnMessage
    public String login(String sessionID, Session session) {
//        you can do this instead of exception and it will still leak memory
//        try {
//            session.close(new CloseReason(CloseCodes.GOING_AWAY, "Clearing session"));
//        } catch (IOException e) {
//            e.printStackTrace();
//        }
        throw new IllegalArgumentException("Testing Leak");
    }

    @OnClose
    public void close(Session session, CloseReason reason) {
    }

    @OnError
    public void error(Session session, Throwable error) {
        error.printStackTrace();
    }

}

Here is the front end html:

<!DOCTYPE html>
<head>
    <script type="text/javascript" src="app.js"></script> </head> <body>
    <div>
        <button onclick="test()">
            LEAK
        </button>
    </div>
</body>
</html>

Here is js file:

    var webSocket = null;

    function test() {
        console.log("test button clicked");
        openSocket();
    }

    function openSocket() {
        console.log("open webSocket");
        webSocket = new WebSocket("ws://" + window.location.host + "/memoryleak/asyncMsg");

        webSocket.onopen = function(event){
            console.log("webSocket open");
            webSocket.send('test');
        };

        webSocket.onmessage = function(event){
            console.log("webSocket onMessage", event);
        };

        webSocket.onclose = function(event){
            webSocket = null;
            console.log("webSocket connection closed", event);
            openSocket();
        };
    }

    function closeSocket(){
        console.log("close webSocket");
        if (webSocket)
            webSocket.close();
    }

You need to run it on multiple clients if you want it leak fast. We did it with 8 clients and it leaked 3GB with in like 10 min.

Thank you
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>

Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314 serge@american-data.com<ma...@american-data.com> | www.american-data.com


[Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>


Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data and contains information that may be confidential and/or privileged. If you are not the intended recipient, you should not use, disclose or take any action based on the message. If you have received this transmission in error, please immediately contact the sender by return e-mail and delete this e-mail, and any attachments, from any computer.

[ECS 10 Conversion]<http://www.american-data.com/viewer>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

Posted by Serge Perepel <se...@american-data.com>.
-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]

>This is likely to get looked at faster if you provide code that other people can run that will reproduce the issue you are seeing rather than expecting someone else to construct the test case for you.

Here is how you can reproduce it:

Server websocket:

package ad.ecs.websocket;

import java.io.IOException;

import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;

@ServerEndpoint(value = "/asyncMsg")
public class TestWebsocket {
    @OnOpen
    public void open(Session session) throws IOException{
        session.getBasicRemote().sendText("Connection Established");
    }

    @OnMessage
    public String login(String sessionID, Session session) {
//        you can do this instead of exception and it will still leak memory
//        try {
//            session.close(new CloseReason(CloseCodes.GOING_AWAY, "Clearing session"));
//        } catch (IOException e) {
//            e.printStackTrace();
//        }
        throw new IllegalArgumentException("Testing Leak");
    }

    @OnClose
    public void close(Session session, CloseReason reason) {
    }

    @OnError
    public void error(Session session, Throwable error) {
        error.printStackTrace();
    }

}

Here is the front end html:

<!DOCTYPE html>
<head>
    <script type="text/javascript" src="app.js"></script>
</head>
<body>
    <div>
        <button onclick="test()">
            LEAK
        </button>
    </div>
</body>
</html>

Here is js file:

    var webSocket = null;

    function test() {
        console.log("test button clicked");
        openSocket();
    }

    function openSocket() {
        console.log("open webSocket");
        webSocket = new WebSocket("ws://" + window.location.host + "/memoryleak/asyncMsg");

        webSocket.onopen = function(event){
            console.log("webSocket open");
            webSocket.send('test');
        };

        webSocket.onmessage = function(event){
            console.log("webSocket onMessage", event);
        };

        webSocket.onclose = function(event){
            webSocket = null;
            console.log("webSocket connection closed", event);
            openSocket();
        };
    }

    function closeSocket(){
        console.log("close webSocket");
        if (webSocket)
            webSocket.close();
    }

You need to run it on multiple clients if you want it leak fast. We did it with 8 clients and it leaked 3GB with in like 10 min.

Thank you
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>

Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
serge@american-data.com<ma...@american-data.com> | www.american-data.com


[Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>


Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are not
the intended recipient, you should not use, disclose or take any action based on
the message. If you have received this transmission in error, please immediately
contact the sender by return e-mail and delete this e-mail,
and any attachments, from any computer.

[ECS 10 Conversion]<http://www.american-data.com/viewer>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

Posted by Mark Thomas <ma...@apache.org>.
On 26/01/18 15:11, Serge Perepel wrote:
> When we run our app for 2-3 hours we experience this leak related to our web sockets connections, here is the Memory Analyzer Suspect:
> 
> One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>".

A break down of that ~2Gb would be helpful. How many entries in the Map?
What is the distribution of sizes?

> Keywords
> org.apache.coyote.AbstractProtocol$ConnectionHandler
> java.util.concurrent.ConcurrentHashMap$Node[]
> java.net.URLClassLoader @ 0x720029098
> 
> 
> Also these bugs might be related to this one:
> 
> 
> 
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57546
> 
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57750
> 
> 
> 
> The 57546 bug looks very similar to what we are experiencing. We tested on Linux and so far we do not see the same behavior. Also this is our web socket code:
> 
> 
> 
> package ad.ecs.async.websocket;
> 
> 
> 
> import java.io.IOException;
> 
> import java.util.Arrays;
> 
> 
> 
> import javax.websocket.CloseReason;
> 
> import javax.websocket.OnClose;
> 
> import javax.websocket.OnError;
> 
> import javax.websocket.OnMessage;
> 
> import javax.websocket.OnOpen;
> 
> import javax.websocket.Session;
> 
> import javax.websocket.server.ServerEndpoint;
> 
> 
> 
> import ad.common.Global;
> 
> import ad.ecs.async.AsyncEngine;
> 
> import ad.ecs.async.AsyncResponse;
> 
> import ad.ecs.async.AsyncType;
> 
> import ad.ecs.db.DatabaseEngine;
> 
> import ad.ecs.db.paradox.User;
> 
> import ad.ecs.security.engine.SecurityEngine;
> 
> 
> 
> /**
> 
>  * @author Serge Perepel
> 
>  * @since Aug 23, 2017 12:23:32 PM
> 
>  */
> 
> @ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)
> 
> public class ECSAsync {
> 
> 
> 
>     @OnOpen
> 
>     public void open(Session session) throws IOException{
> 
>         session.getBasicRemote().sendText("Connection Established");
> 
>     }
> 
> 
> 
>     @OnMessage
> 
>     public String login(String sessionID, Session session) {
> 
>         AsyncEngine.INSTANCE.wsConnect(session, sessionID);
> 
>         org.hibernate.Session dbSession = DatabaseEngine.getSessionFactory().openSession();
> 
>         try {
> 
>                int userID = SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);
> 
>                User user = (User) dbSession.get(User.class, userID);
> 
>                if (user != null) {
> 
>                        if (user.getNextLogin() == 1) {
> 
>                                AsyncResponse response = new AsyncResponse();
> 
>                                response.setType(AsyncType.None);
> 
>                                response.setData(Arrays.asList("PASSWORD"));
> 
>                                response.setObjData("PASSWORD");
> 
>                                AsyncEngine.INSTANCE.addTransientResult(sessionID, response);
> 
>                        }
> 
>                }
> 
>         } finally {
> 
>                dbSession.close();
> 
>         }
> 
>         return "ok";
> 
>     }
> 
> 
> 
>     @OnClose
> 
>     public void close(Session session, CloseReason reason) {
> 
>         AsyncEngine.INSTANCE.wsDisconnect(session);
> 
>     }
> 
> 
> 
>     @OnError
> 
>     public void error(Session session, Throwable error) {
> 
>         Global.INSTANCE.getLogHelper().exception(error);
> 
>         //session.close(new CloseReason(closeCode, reasonPhrase));
> 
>     }
> 
> }
> 
> 
> 
> Front end opens connection and sends invalid sessionID which causes
> 
> the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw exception and after that disconnect happens on the front end. At this point front end opens new connection and process goes into the loop. I'm assuming that the connection handler suppose to get freed after a disconnect happen. But it seems to accumulate. You can try this code and just replace code in the login method to always throw the exception. On the front end onDisconnect try to open new connection and send a random message to the web socket.

This is likely to get looked at faster if you provide code that other
people can run that will reproduce the issue you are seeing rather than
expecting someone else to construct the test case for you.

Mark


> 
> 
> 
> Thank you
> 
> Serge
> 
> [http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>
> 
> Serge Perepel
> Software Developer | American Data
> p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
> serge@american-data.com<ma...@american-data.com> | www.american-data.com
> 
> 
> [Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>
> 
> 
> Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
> and contains information that may be confidential and/or privileged. If you are not
> the intended recipient, you should not use, disclose or take any action based on
> the message. If you have received this transmission in error, please immediately
> contact the sender by return e-mail and delete this e-mail,
> and any attachments, from any computer.
> 
> [ECS 10 Conversion]<http://www.american-data.com/viewer>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

Posted by Serge Perepel <se...@american-data.com>.
Forgot to mention that we are running Tomcat 8.5 on Windows 2012 Server

-----Original Message-----
From: Serge Perepel [mailto:serge@american-data.com] 
Sent: Friday, January 26, 2018 9:12 AM
To: users@tomcat.apache.org
Subject: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket

When we run our app for 2-3 hours we experience this leak related to our web sockets connections, here is the Memory Analyzer Suspect:

One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>".

Keywords
org.apache.coyote.AbstractProtocol$ConnectionHandler
java.util.concurrent.ConcurrentHashMap$Node[]
java.net.URLClassLoader @ 0x720029098


Also these bugs might be related to this one:



https://bz.apache.org/bugzilla/show_bug.cgi?id=57546

https://bz.apache.org/bugzilla/show_bug.cgi?id=57750



The 57546 bug looks very similar to what we are experiencing. We tested on Linux and so far we do not see the same behavior. Also this is our web socket code:



package ad.ecs.async.websocket;



import java.io.IOException;

import java.util.Arrays;



import javax.websocket.CloseReason;

import javax.websocket.OnClose;

import javax.websocket.OnError;

import javax.websocket.OnMessage;

import javax.websocket.OnOpen;

import javax.websocket.Session;

import javax.websocket.server.ServerEndpoint;



import ad.common.Global;

import ad.ecs.async.AsyncEngine;

import ad.ecs.async.AsyncResponse;

import ad.ecs.async.AsyncType;

import ad.ecs.db.DatabaseEngine;

import ad.ecs.db.paradox.User;

import ad.ecs.security.engine.SecurityEngine;



/**

 * @author Serge Perepel

 * @since Aug 23, 2017 12:23:32 PM

 */

@ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)

public class ECSAsync {



    @OnOpen

    public void open(Session session) throws IOException{

        session.getBasicRemote().sendText("Connection Established");

    }



    @OnMessage

    public String login(String sessionID, Session session) {

        AsyncEngine.INSTANCE.wsConnect(session, sessionID);

        org.hibernate.Session dbSession = DatabaseEngine.getSessionFactory().openSession();

        try {

               int userID = SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);

               User user = (User) dbSession.get(User.class, userID);

               if (user != null) {

                       if (user.getNextLogin() == 1) {

                               AsyncResponse response = new AsyncResponse();

                               response.setType(AsyncType.None);

                               response.setData(Arrays.asList("PASSWORD"));

                               response.setObjData("PASSWORD");

                               AsyncEngine.INSTANCE.addTransientResult(sessionID, response);

                       }

               }

        } finally {

               dbSession.close();

        }

        return "ok";

    }



    @OnClose

    public void close(Session session, CloseReason reason) {

        AsyncEngine.INSTANCE.wsDisconnect(session);

    }



    @OnError

    public void error(Session session, Throwable error) {

        Global.INSTANCE.getLogHelper().exception(error);

        //session.close(new CloseReason(closeCode, reasonPhrase));

    }

}



Front end opens connection and sends invalid sessionID which causes

the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw exception and after that disconnect happens on the front end. At this point front end opens new connection and process goes into the loop. I'm assuming that the connection handler suppose to get freed after a disconnect happen. But it seems to accumulate. You can try this code and just replace code in the login method to always throw the exception. On the front end onDisconnect try to open new connection and send a random message to the web socket.



Thank you

Serge

[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]<www.american-data.com>

Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314 serge@american-data.com<ma...@american-data.com> | www.american-data.com


[Follow us on Facebook]<https://www.facebook.com/pages/American-Data/100445186666233>   [Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>    [Follow us on Twitter] <https://twitter.com/AmericanDataECS>   [Follow us on Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>


Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data and contains information that may be confidential and/or privileged. If you are not the intended recipient, you should not use, disclose or take any action based on the message. If you have received this transmission in error, please immediately contact the sender by return e-mail and delete this e-mail, and any attachments, from any computer.

[ECS 10 Conversion]<http://www.american-data.com/viewer>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org