You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Ildar Absalyamov <il...@gmail.com> on 2016/02/22 02:02:21 UTC

Socket feed adapter reconnect policy

Hi all,

Whenever the socket-based feed adapter is used, what happens when the socket is closed on the client side?
Is it supposed to continue ingestion once the connection is reestablished again? 
Did anyone tried that? From my observations when the connection is established the second time, on the feed’s side socket read operation blocks and further ingestion does not happen.

Best regards,
Ildar

Re: Socket feed adapter reconnect policy

Posted by Ildar Absalyamov <il...@gmail.com>.
Abdullah,

This is the feed configuration that I am using:

drop dataverse experiments if exists;
create dataverse experiments;
use dataverse experiments;
create type TwitterUserType as closed {
    screen-name: string,
    lang: string,
    friends_count: int32,
    statuses_count: int32,
    name: string,
    followers_count: int32
}
create type TweetMessageType as closed {
    tweetid: int64,
    user: TwitterUserType,
    sender-location: point,
    send-time: datetime,
    referred-topics: {{ string }},
    message-text: string
}
create dataset Tweets(TweetMessageType) primary key tweetid;
create feed TweetFeed1 using socket_adapter
(
    ("sockets"="127.0.0.1:10001"),
    ("address-type"="IP"),
    ("type-name"="TweetMessageType"),
    ("format"="adm"),
    ("duration"="1200")
);
set wait-for-completion-feed "false";
connect feed TweetFeed1 to dataset Tweets;

After that I start pouring tweet records via socket 127.0.0.1:10001. After the first batch is ingested (does no matter what the size is) record generator stops and closes socket from the client side. Then I am launching record generator again via socket (note feed is still connected) and that’s where ingestion stops.

You can check the modified Young-Seek’s ingestion clients which I am using https://github.com/ty1er/asterix-statistics-experiments. I can give you tips how to start it. 

> On Feb 21, 2016, at 20:13, abdullah alamoudi <ba...@gmail.com> wrote:
> 
> Ildar,
> Can you share an example I can try and work on?
> 
> I have created a "test" change over the weekend that allows me to start
> different servers and test the socket/RSS feeds on them. unfortunately
> waiting for other reviews for other fixes to make it before I can add the
> new test cases.
> 
> I would fix this issue and add a test case for it but I am not clear on
> what is happening vs. the expected behavior. More details would help us fix
> this.
> Thanks,
> Abdullah.
> 
> On Mon, Feb 22, 2016 at 4:02 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
> 
>> Hi all,
>> 
>> Whenever the socket-based feed adapter is used, what happens when the
>> socket is closed on the client side?
>> Is it supposed to continue ingestion once the connection is reestablished
>> again?
>> Did anyone tried that? From my observations when the connection is
>> established the second time, on the feed’s side socket read operation
>> blocks and further ingestion does not happen.
>> 
>> Best regards,
>> Ildar
>> 
>> 

Best regards,
Ildar

Re: Socket feed adapter reconnect policy

Posted by abdullah alamoudi <ba...@gmail.com>.
Ildar,
Can you share an example I can try and work on?

I have created a "test" change over the weekend that allows me to start
different servers and test the socket/RSS feeds on them. unfortunately
waiting for other reviews for other fixes to make it before I can add the
new test cases.

I would fix this issue and add a test case for it but I am not clear on
what is happening vs. the expected behavior. More details would help us fix
this.
Thanks,
Abdullah.

On Mon, Feb 22, 2016 at 4:02 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> Hi all,
>
> Whenever the socket-based feed adapter is used, what happens when the
> socket is closed on the client side?
> Is it supposed to continue ingestion once the connection is reestablished
> again?
> Did anyone tried that? From my observations when the connection is
> established the second time, on the feed’s side socket read operation
> blocks and further ingestion does not happen.
>
> Best regards,
> Ildar
>
>