[Catalina] Implementing other protocols

[Serle Shuman <se...@creator.co.za>]

I've been trying to implement an FTP protocol under Catalina and have experienced the following issues:
Implementing FtpConnector,FtpRequest, FtpResponse and their underlying implementation is not an issue.
I am having trouble with the container mapping, i.e 
    the standardengine class finds the appropriate host
    the standardhost class then gets stuck finding the appropriate context.

I think what is needed here is for the standard host class to look at the protocol property of the request and then to find a StandardProtocol class (and all the great interfaces) which then would find (in a protocol specific manner) the appropriate context. We could then also clean StandardHost to never assume HttpRequests and HttpResponses. i.e we could correctly make this assumption only from the protocol layer up. 

This would be really neat for FTP where the protocol layer could be responsible for context negotiation, eventually handing over to a StandardFTPServlet within a given context at the higher level. This could be implemented by the StandardProtocol containing StandardProtocolValves.

I think the main points which are generally applicable to implementing other protocols under catalina are:
Protocal specific context mapping;
Protocol specific Servlets (have not given this much thought yet)  

Could I have comment on these points

Regards
Serle

[Catalina] HTTP/1.1 connector proposal

[Remy Maucherat <re...@apache.org>]

Here are two proposals. The first one is a modification to the Realm
interface, and the other describes the proposed architecture of the HTTP/1.1
connector. Comments / suggestions / ideas wanted :-)

- A getPassword() should be added to the realm interface. It's needed to
support the HTTP DIGEST authentication. Alternatively, a getDigest() method
would work, but then each realm would have to know how to calculate the
digest, which is a bad thing (IMO). Unless, of course, we provide a default
implementation for it in an AbstractRealm class.

- HTTP/1.1

First of all, what's wrong, performance wise, with the current connector ?
I had a quick look at it; it has a thread pool (which seems fine to me) to
process the requests, it recycles all its stuff. The request headers parsing
could be optimized, but that's about it, I think.

The goal of the new connector is to support :
- Persistent connections
- Pipelining
- Content chunking
- Entity body coding (in the non-RFC world, it's called data compression ;-)
- Use 100 - Continue with pipelining
- Connection header support (to know when to close the connection, or to
signal that the connection is about to be closed by the server)

Additional modifications in other parts of Catalina are needed, mainly in
the DefaultServlet, to handle the HTTP/1.1 specific headers, like the ones
used for resuming, the charset (already done, I think), etag ... I'll focus
on those later on, as they are slight modifications when compared to the
ones which have to be made in the architecture.

I plan to use the same kind of thread pool for handling processors.

Each processor will have a flag indicating its status : idle, parsing,
executing. This could be useful when the connector wants to force the reuse
of one of the processors.
As suggested above, the processor will use the content length information
included with each request (or use chunks length) to know when a request
entity-body will end. The connection is not closed when the processor has
finished reading a request, unless it is specifically requested by the
client, and the processor goes on with the next pipelined request.

The processor will encapsulate the InputStream in an object derived from
BufferedInputStream. This stream will prevent the "client" servlet from
directly having control over the connection (like closing if, for example).
It also has to prevent the client from consuming more bytes that what is
specified in the content length or by the chunking.

The BufferedInput(Output)Stream derived object will also have to check and
handle chunking and content coding of the entity-body. In the case of
chunking, that means that at least a whole chunk has to be read each time,
which translates into a variable length buffer. I'd like to avoid
reallocating a new buffer all the time, but I can't figure out how. Ideas ?

I propose naming those Input/OutputStreams HttpServletInput(Output)Stream
(or HttpInput(Output)StreamFacade ?).

Additionally, when pipelining, each time a request header block is
successfully
parsed, the processor will notify the client by sending a 100 - Continue
reply.

I will hopefully be able to complete a first draft implementation for most
of this tomorrow.

Remy

Re: [Catalina] Implementing other protocols

[Serle Shuman <se...@creator.co.za>]

I'm not really sure what constitutes  a request/response protocol but FTP
sends a command to the server (request) and the client recieves a commond
confirmation (response), at least on the command channel. As for the data
channel?

I would see a host roughly equating to a customer's internet presence. I
would therefore like to configure and manage protocol specific services on a
per customer basis in one place. I see this in the same light as the context
concept introduced by the Servlet 2.2 spec for web applications.

In my case the FTP security and processing needs to be managed on a per
customer basis, however this tends to be closely coupled to the web-app
side. Infact my FTP processing would re-use underlying classes contained in
the client's existing web-app. Hence I would prefer to have a logical
mapping between host<->customer. Your mapping approach seems like the way to
go.

Some time ago there was a discussion about Servlets for other potocols, were
these limited to protocols ontop of HTTP e.g HTM, XML, WML etc. or was
throught given to other protocols e.g. FTP, TELNET SMTP etc.

Serle

----- Original Message -----
From: Craig R. McClanahan <Cr...@eng.sun.com>
To: <to...@jakarta.apache.org>
Sent: Thursday, May 18, 2000 7:21 PM
Subject: Re: [Catalina] Implementing other protocols


> Serle Shuman wrote:
>
> > I've been trying to implement an FTP protocol under Catalina and have
experienced the following issues:
> > Implementing FtpConnector,FtpRequest, FtpResponse and their underlying
implementation is not an issue.
> > I am having trouble with the container mapping, i.e
> >     the standardengine class finds the appropriate host
> >     the standardhost class then gets stuck finding the appropriate
context.
>
> Does FTP really fit the request/response processing model?  I would be
interested in hearing more about what your thinking is here -- is this a
general purpose FTP server, or an alternative to using HTP for file
upload/download processing.
>
> >
> > I think what is needed here is for the standard host class to look at
the protocol property of the request and then to find a StandardProtocol
class (and all the great interfaces) which then would find (in a protocol
specific manner) the appropriate context. We could then also clean
StandardHost to never assume HttpRequests and HttpResponses. i.e we could
correctly make this assumption only from the protocol layer up.
> >
> > This would be really neat for FTP where the protocol layer could be
responsible for context negotiation, eventually handing over to a
StandardFTPServlet within a given context at the higher level. This could be
implemented by the StandardProtocol containing StandardProtocolValves.
> >
>
> I can think of at least two ways to do this:
>
> * An alternate implementation of Host that does the map() call differently
>   for each protocol (which is the only place HTTP is assumed in
StandardHost).
>   You could then make the per-protocol decision at the engine level
instead
>   (or just configure a complete container hierarchy per protocol since
they
>   usually listen on different ports anyway).
>
> * An abstraction layer for doing "mapping" that could be selected
>   on a protocol-specific basis.  In general, an implementation of map
>   would do two things:
>     -    Examine the request properties to select the
>         appropriate child container (this works at all
>         levels of the containment hierarchy, not just
>         Host->Context).
>     -    Optionally update the request properties to
>         reflect the mapping choices.  (For example,
>         StandardHost calls setContext() to tell the
>         request which Context was selected.)
>
> The second approach sounds most like what you are talking about, right?
My only concern about it is that you add a little extra runtime overhead on
every request (hashtable lookup on the protocol to get the right Mapper).
On the other hand, this lets you have multiprotocol support in the same
Host, which you cannot do if you mandate separate Hosts per protocol.
>
> >
> > I think the main points which are generally applicable to implementing
other protocols under catalina are:
> > Protocal specific context mapping;
> > Protocol specific Servlets (have not given this much thought yet)
>
> Either of the approaches described above would deal with this
(Context->Wrapper selection) as well.  It is the same basic issue, one level
lower in terms of detail.
>
> >
> > Could I have comment on these points
>
> I wonder how likely it is that people would want multi-protocol support in
a single Container.  If this is rare, I'd rather not increase the overhead,
and just have the sysadmin configure a separate Container stack for each
protocol.  If it's common, then the mapping abstraction seems like a pretty
straightforward way to deal with it.
>
> >
> > Regards
> > Serle
>
> Craig
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org
>

Re: [Catalina] Implementing other protocols

["Craig R. McClanahan" <Cr...@eng.sun.com>]

Serle Shuman wrote:

> I've been trying to implement an FTP protocol under Catalina and have experienced the following issues:
> Implementing FtpConnector,FtpRequest, FtpResponse and their underlying implementation is not an issue.
> I am having trouble with the container mapping, i.e
>     the standardengine class finds the appropriate host
>     the standardhost class then gets stuck finding the appropriate context.

Does FTP really fit the request/response processing model?  I would be interested in hearing more about what your thinking is here -- is this a general purpose FTP server, or an alternative to using HTP for file upload/download processing.

>
> I think what is needed here is for the standard host class to look at the protocol property of the request and then to find a StandardProtocol class (and all the great interfaces) which then would find (in a protocol specific manner) the appropriate context. We could then also clean StandardHost to never assume HttpRequests and HttpResponses. i.e we could correctly make this assumption only from the protocol layer up.
>
> This would be really neat for FTP where the protocol layer could be responsible for context negotiation, eventually handing over to a StandardFTPServlet within a given context at the higher level. This could be implemented by the StandardProtocol containing StandardProtocolValves.
>

I can think of at least two ways to do this:

* An alternate implementation of Host that does the map() call differently
  for each protocol (which is the only place HTTP is assumed in StandardHost).
  You could then make the per-protocol decision at the engine level instead
  (or just configure a complete container hierarchy per protocol since they
  usually listen on different ports anyway).

* An abstraction layer for doing "mapping" that could be selected
  on a protocol-specific basis.  In general, an implementation of map
  would do two things:
    -    Examine the request properties to select the
        appropriate child container (this works at all
        levels of the containment hierarchy, not just
        Host->Context).
    -    Optionally update the request properties to
        reflect the mapping choices.  (For example,
        StandardHost calls setContext() to tell the
        request which Context was selected.)

The second approach sounds most like what you are talking about, right?  My only concern about it is that you add a little extra runtime overhead on every request (hashtable lookup on the protocol to get the right Mapper).  On the other hand, this lets you have multiprotocol support in the same Host, which you cannot do if you mandate separate Hosts per protocol.

>
> I think the main points which are generally applicable to implementing other protocols under catalina are:
> Protocal specific context mapping;
> Protocol specific Servlets (have not given this much thought yet)

Either of the approaches described above would deal with this (Context->Wrapper selection) as well.  It is the same basic issue, one level lower in terms of detail.

>
> Could I have comment on these points

I wonder how likely it is that people would want multi-protocol support in a single Container.  If this is rare, I'd rather not increase the overhead, and just have the sysadmin configure a separate Container stack for each protocol.  If it's common, then the mapping abstraction seems like a pretty straightforward way to deal with it.

>
> Regards
> Serle

Craig