You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Jianfeng Jia <ji...@gmail.com> on 2016/01/04 22:46:28 UTC
Re: Need Feed experts' help with an hanging issue
@Abdullah, any luck on this issue? Thanks and happy new year!
> On Nov 30, 2015, at 5:24 PM, abdullah alamoudi <ba...@gmail.com> wrote:
>
> I know exactly what is going on here. The problem is you pointed out is
> caused by the duplicate keys. If I remember correctly, the main issue is
> that locks that are placed on the primary keys are not released.
>
> I will start fixing this issue tonight.
> Cheers,
> Abdullah.
>
> Amoudi, Abdullah.
>
> On Mon, Nov 30, 2015 at 4:52 PM, Jianfeng Jia <ji...@gmail.com>
> wrote:
>
>> Dear devs,
>>
>> I hit an wield issue that is reproducible, but only if the data has
>> duplications and also is large enough. Let me explained it step by step:
>>
>> 1. The dataset is very simple that only has two fields.
>> DDL AQL:
>> —————————————
>> drop dataverse test if exists;
>> create dataverse test;
>> use dataverse test;
>>
>> create type t_test as closed{
>> fa: int64,
>> fb : int64
>> }
>>
>> create dataset ds_test(t_test) primary key fa;
>>
>> create feed fd_test using socket_adapter
>> (
>> ("sockets"="nc1:10001"),
>> ("address-type"="nc"),
>> ("type-name"="t_test"),
>> ("format"="adm"),
>> ("duration"="1200")
>> );
>>
>> set wait-for-completion-feed "false";
>> connect feed fd_test to dataset ds_test using policy AdvancedFT_Discard;
>>
>> ——————————————————————————————
>>
>> That AdvancedFT_Discard policy will ignore the exception from the
>> insertion and keep ingesting.
>>
>> 2. Ingesting the data by a very simple socked adapter which reads the
>> record one by one from an adm file. The src is here:
>> https://github.com/JavierJia/twitter-tracker/blob/master/src/main/java/edu/uci/ics/twitter/asterix/feed/FileFeedSocketAdapterClient.java
>> The data and the app package is provided here:
>> https://drive.google.com/folderview?id=0B423M7wGZj9dYVQ1TkpBNzcwSlE&usp=sharing
>> To feed the data you can run:
>>
>> ./bin/feedFile -u 172.17.0.2 -p 10001 -c 5000000 ~/data/twitter/test.adm
>>
>> -u for sever url
>> -p for server port
>> -c for count of line you want to ingest
>>
>> 3. After ingestion, all the requests about the ds_test was hanging. There
>> is no exception and no responds for hours. However it can respond any other
>> queries that on other datasets, like Metadata.
>>
>> That data contains some duplicated records which should trigger the insert
>> exception. If I change the count from 5000000 to lower, let’s say 3000000,
>> it has no problems, although it contains duplications as well.
>>
>> Any feed experts have any hint on which part could be wrong? cc and nc log
>> was attached. Thank you!
>>
>>
>>
>>
>>
>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>>
>>
Best,
Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine
Re: Need Feed experts' help with an hanging issue
Posted by abdullah alamoudi <ba...@gmail.com>.
Hi Jianfeng,
Not yet. I would say sometime within the next two days but to be safe, I
will pump that to sometimes within this week :)
Will let you know the moment it is fixed.
Cheers and happy new year,
Abdullah.
Amoudi, Abdullah.
On Tue, Jan 5, 2016 at 12:46 AM, Jianfeng Jia <ji...@gmail.com>
wrote:
> @Abdullah, any luck on this issue? Thanks and happy new year!
>
> > On Nov 30, 2015, at 5:24 PM, abdullah alamoudi <ba...@gmail.com>
> wrote:
> >
> > I know exactly what is going on here. The problem is you pointed out is
> > caused by the duplicate keys. If I remember correctly, the main issue is
> > that locks that are placed on the primary keys are not released.
> >
> > I will start fixing this issue tonight.
> > Cheers,
> > Abdullah.
> >
> > Amoudi, Abdullah.
> >
> > On Mon, Nov 30, 2015 at 4:52 PM, Jianfeng Jia <ji...@gmail.com>
> > wrote:
> >
> >> Dear devs,
> >>
> >> I hit an wield issue that is reproducible, but only if the data has
> >> duplications and also is large enough. Let me explained it step by step:
> >>
> >> 1. The dataset is very simple that only has two fields.
> >> DDL AQL:
> >> —————————————
> >> drop dataverse test if exists;
> >> create dataverse test;
> >> use dataverse test;
> >>
> >> create type t_test as closed{
> >> fa: int64,
> >> fb : int64
> >> }
> >>
> >> create dataset ds_test(t_test) primary key fa;
> >>
> >> create feed fd_test using socket_adapter
> >> (
> >> ("sockets"="nc1:10001"),
> >> ("address-type"="nc"),
> >> ("type-name"="t_test"),
> >> ("format"="adm"),
> >> ("duration"="1200")
> >> );
> >>
> >> set wait-for-completion-feed "false";
> >> connect feed fd_test to dataset ds_test using policy AdvancedFT_Discard;
> >>
> >> ——————————————————————————————
> >>
> >> That AdvancedFT_Discard policy will ignore the exception from the
> >> insertion and keep ingesting.
> >>
> >> 2. Ingesting the data by a very simple socked adapter which reads the
> >> record one by one from an adm file. The src is here:
> >>
> https://github.com/JavierJia/twitter-tracker/blob/master/src/main/java/edu/uci/ics/twitter/asterix/feed/FileFeedSocketAdapterClient.java
> >> The data and the app package is provided here:
> >>
> https://drive.google.com/folderview?id=0B423M7wGZj9dYVQ1TkpBNzcwSlE&usp=sharing
> >> To feed the data you can run:
> >>
> >> ./bin/feedFile -u 172.17.0.2 -p 10001 -c 5000000 ~/data/twitter/test.adm
> >>
> >> -u for sever url
> >> -p for server port
> >> -c for count of line you want to ingest
> >>
> >> 3. After ingestion, all the requests about the ds_test was hanging.
> There
> >> is no exception and no responds for hours. However it can respond any
> other
> >> queries that on other datasets, like Metadata.
> >>
> >> That data contains some duplicated records which should trigger the
> insert
> >> exception. If I change the count from 5000000 to lower, let’s say
> 3000000,
> >> it has no problems, although it contains duplications as well.
> >>
> >> Any feed experts have any hint on which part could be wrong? cc and nc
> log
> >> was attached. Thank you!
> >>
> >>
> >>
> >>
> >>
> >>
> >> Best,
> >>
> >> Jianfeng Jia
> >> PhD Candidate of Computer Science
> >> University of California, Irvine
> >>
> >>
> >>
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>