Best way to sniffer thrift requests

[郑华斌 <hu...@qq.com>]

For service S, it has N apis, what's the best way to copy requests of specified apis and resend them to a message queue, without affecting the normal service?


Maybe there are following possible ways:
1) tcp copy: with tcp copy tools that copy requests on the server side, and redirect the copied flow to a fake server, there requests of different apis are send to message queue, in the format of api:serialized_request. But deploying is difficult because tcpcopy requires root permission.
2) develop a transport to wrapped current transport on server side, and send buffers to mq as well as call underline transport. But how to identify which api a request buffer belong to?


Any ideas?

Re: Best way to sniffer thrift requests

[Randy Abernethy <ra...@gmail.com>]

Hi Huabin,

I might attack the problem at a higher level and define the API in terms of
messages, structs in Apache Thrift. Then when a server receives a requests
in the form of of a struct, it can serialize the struct to a memory buffer
and send the buffer out as a message queue message.

Apache Thrift makes this very easy because all structs have read() and
write() methods that can be used to serialize them to/from any I/O stack.
So if you create a TMemoryBuffer transport and add a TCompactProtocol to it
you can write your struct out to the mem buf and then email it (or
whatever).

Here's an example in Python but the concept is the same with any language:

https://github.com/RandyAbernethy/ThriftBook/blob/master/part3/mq/QuoteGen.py

If you really need a low level solution, I would using message framing and
then create a custom framing layer that forks a copy of every frame out to
the message queue while passing another copy up the stack to the protocol
(e.g. TCompactProtocol/TMyFrameDupTrans/TSocket). There's a custom tee
transport example in Java here:

https://github.com/RandyAbernethy/ThriftBook/blob/master/part2/servers/factories/TTeeTransport.java

Hope this helps.

Best,
Randy


On Tue, Jul 11, 2017 at 2:16 AM, 郑华斌 <hu...@qq.com> wrote:

> For service S, it has N apis, what's the best way to copy requests of
> specified apis and resend them to a message queue, without affecting the
> normal service?
>
>
> Maybe there are following possible ways:
> 1) tcp copy: with tcp copy tools that copy requests on the server side,
> and redirect the copied flow to a fake server, there requests of different
> apis are send to message queue, in the format of api:serialized_request.
> But deploying is difficult because tcpcopy requires root permission.
> 2) develop a transport to wrapped current transport on server side, and
> send buffers to mq as well as call underline transport. But how to identify
> which api a request buffer belong to?
>
>
> Any ideas?