Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[mingda li <li...@gmail.com>]

Hi,

Recently, I am trying to run the data feed query on a cluster of 16 nodes.
All the query run well without error. But each time after I disconnect the
feed, there is none of tweet data stored in dataverse. However, when I try
to run other AQL queries, the cluster can work well. And I have ever used 4
nodes cluster to load Tweet data successfully. I also checked the log file
and find no error there. This is wired. Does anyone know why? Has anyone
ever used the data feed function on a cluster of 16 nodes or more?

Bests,
Mingda

Re: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[Wail Alkowaileet <wa...@gmail.com>]

Hi Mingda,

Just out of curiosity can you load somehow "large" dataset and query it on
the 16 nodes (e.g query the number of records in the dataset). I want to
make sure all NCs are participating.
If that succeed, probably you should try to load the same data using a
localfs feed and run query after the ingestion finishes.

This will help us to isolate the problem if it's a twitter feed problem or
something else.


On Fri, Oct 14, 2016 at 10:13 AM, Xikui Wang <xi...@uci.edu> wrote:

> Actually we have talked, and I don't have clear answer. The current twitter
> adaptor is getting node from (IExternalDataSourceFactory)
> AsterixAppContextInfo.INSTANCE.getMetadataProperties().
> getStores().keySet()
> and select the first partition from that node based on
> ClusterStateManager.INSTANCE.getIODevices(node).length
>
> The partition constraints of intake adaptor will only have the first node,
> 'nc1' in local case. Meaning it will run on the first node only. It's not
> obvious to me why it's working on 4 nodes but not 16 nodes...
>
> Best,
> Xikui
>
> On Thu, Oct 13, 2016 at 11:16 PM, Mike Carey <dt...@gmail.com> wrote:
>
> > +dev
> >
> > @Xikui:  Can you help Mingda?
> >
> >
> >
> > On 10/13/16 7:45 PM, mingda li wrote:
> >
> >> Hi,
> >>
> >> Recently, I am trying to run the data feed query on a cluster of 16
> >> nodes. All the query run well without error. But each time after I
> >> disconnect the feed, there is none of tweet data stored in dataverse.
> >> However, when I try to run other AQL queries, the cluster can work well.
> >> And I have ever used 4 nodes cluster to load Tweet data successfully. I
> >> also checked the log file and find no error there. This is wired. Does
> >> anyone know why? Has anyone ever used the data feed function on a
> cluster
> >> of 16 nodes or more?
> >>
> >> Bests,
> >> Mingda
> >>
> >
> >
>



-- 

*Regards,*
Wail Alkowaileet

Re: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[Xikui Wang <xi...@uci.edu>]

Actually we have talked, and I don't have clear answer. The current twitter
adaptor is getting node from (IExternalDataSourceFactory)
AsterixAppContextInfo.INSTANCE.getMetadataProperties().getStores().keySet()
and select the first partition from that node based on
ClusterStateManager.INSTANCE.getIODevices(node).length

The partition constraints of intake adaptor will only have the first node,
'nc1' in local case. Meaning it will run on the first node only. It's not
obvious to me why it's working on 4 nodes but not 16 nodes...

Best,
Xikui

On Thu, Oct 13, 2016 at 11:16 PM, Mike Carey <dt...@gmail.com> wrote:

> +dev
>
> @Xikui:  Can you help Mingda?
>
>
>
> On 10/13/16 7:45 PM, mingda li wrote:
>
>> Hi,
>>
>> Recently, I am trying to run the data feed query on a cluster of 16
>> nodes. All the query run well without error. But each time after I
>> disconnect the feed, there is none of tweet data stored in dataverse.
>> However, when I try to run other AQL queries, the cluster can work well.
>> And I have ever used 4 nodes cluster to load Tweet data successfully. I
>> also checked the log file and find no error there. This is wired. Does
>> anyone know why? Has anyone ever used the data feed function on a cluster
>> of 16 nodes or more?
>>
>> Bests,
>> Mingda
>>
>
>

Re: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[Mike Carey <dt...@gmail.com>]

+dev

@Xikui:  Can you help Mingda?


On 10/13/16 7:45 PM, mingda li wrote:
> Hi,
>
> Recently, I am trying to run the data feed query on a cluster of 16 
> nodes. All the query run well without error. But each time after I 
> disconnect the feed, there is none of tweet data stored in dataverse. 
> However, when I try to run other AQL queries, the cluster can work 
> well. And I have ever used 4 nodes cluster to load Tweet data 
> successfully. I also checked the log file and find no error there. 
> This is wired. Does anyone know why? Has anyone ever used the data 
> feed function on a cluster of 16 nodes or more?
>
> Bests,
> Mingda

Re: Re: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[李文海 <lw...@whu.edu.cn>]

Xikui,
Ok, many thanks.

At2016-10-14 12:16:35,李文海<lw...@whu.edu.cn>wrote:
> @Wenhai,
> 
> I think that error only appears in socket feed test cases? In most of the
> socket test cases, the socket server port are set to '10001'.
> 
> Best,
> Xikui
> 
> On Thu, Oct 13, 2016 at 9:06 PM, 李文海 <lw...@whu.edu.cn> wrote:
> 
> >
> > Similar issue arose in our cluster. BTW, we got a series of feed error in
> > a server while running
> > the executionTest in the eclipse. Is there port competition between our
> > feed and the other
> > application? what's the default port of the feed?
> > Best
> >
> >
> >
> > > -----Original Messages-----
> > > From: "mingda li" <li...@gmail.com>
> > > Sent Time: Friday, October 14, 2016
> > > To: dev@asterixdb.apache.org
> > > Cc:
> > > Subject: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while
> > succeed on 4 nodes)
> > >
> > > Hi,
> > >
> > > Recently, I am trying to run the data feed query on a cluster of 16
> > nodes.
> > > All the query run well without error. But each time after I disconnect
> > the
> > > feed, there is none of tweet data stored in dataverse. However, when I
> > try
> > > to run other AQL queries, the cluster can work well. And I have ever
> > used 4
> > > nodes cluster to load Tweet data successfully. I also checked the log
> > file
> > > and find no error there. This is wired. Does anyone know why? Has anyone
> > > ever used the data feed function on a cluster of 16 nodes or more?
> > >
> > > Bests,
> > > Mingda
> >
> >

Re: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[Xikui Wang <xi...@uci.edu>]

@Wenhai,

I think that error only appears in socket feed test cases? In most of the
socket test cases, the socket server port are set to '10001'.

Best,
Xikui

On Thu, Oct 13, 2016 at 9:06 PM, 李文海 <lw...@whu.edu.cn> wrote:

>
> Similar issue arose in our cluster. BTW, we got a series of feed error in
> a server while running
> the executionTest in the eclipse. Is there port competition between our
> feed and the other
> application? what's the default port of the feed?
> Best
>
>
>
> > -----Original Messages-----
> > From: "mingda li" <li...@gmail.com>
> > Sent Time: Friday, October 14, 2016
> > To: dev@asterixdb.apache.org
> > Cc:
> > Subject: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while
> succeed on 4 nodes)
> >
> > Hi,
> >
> > Recently, I am trying to run the data feed query on a cluster of 16
> nodes.
> > All the query run well without error. But each time after I disconnect
> the
> > feed, there is none of tweet data stored in dataverse. However, when I
> try
> > to run other AQL queries, the cluster can work well. And I have ever
> used 4
> > nodes cluster to load Tweet data successfully. I also checked the log
> file
> > and find no error there. This is wired. Does anyone know why? Has anyone
> > ever used the data feed function on a cluster of 16 nodes or more?
> >
> > Bests,
> > Mingda
>
>

Re: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[李文海 <lw...@whu.edu.cn>]

Similar issue arose in our cluster. BTW, we got a series of feed error in a server while running 
the executionTest in the eclipse. Is there port competition between our feed and the other
application? what's the default port of the feed?
Best



> -----Original Messages-----
> From: "mingda li" <li...@gmail.com>
> Sent Time: Friday, October 14, 2016
> To: dev@asterixdb.apache.org
> Cc: 
> Subject: Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)
> 
> Hi,
> 
> Recently, I am trying to run the data feed query on a cluster of 16 nodes.
> All the query run well without error. But each time after I disconnect the
> feed, there is none of tweet data stored in dataverse. However, when I try
> to run other AQL queries, the cluster can work well. And I have ever used 4
> nodes cluster to load Tweet data successfully. I also checked the log file
> and find no error there. This is wired. Does anyone know why? Has anyone
> ever used the data feed function on a cluster of 16 nodes or more?
> 
> Bests,
> Mingda

Fwd: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[mingda li <li...@gmail.com>]

Hi,

Recently, I am trying to run the data feed query on a cluster of 16 nodes.
All the query run well without error. But each time after I disconnect the
feed, there is none of tweet data stored in dataverse. However, when I try
to run other AQL queries, the cluster can work well. And I have ever used 4
nodes cluster to load Tweet data successfully. I also checked the log file
and find no error there. This is wired. Does anyone know why? Has anyone
ever used the data feed function on a cluster of 16 nodes or more?

Bests,
Mingda

Re: Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)

[Mike Carey <dt...@gmail.com>]

+dev

@Xikui:  Can you help Mingda?


On 10/13/16 7:45 PM, mingda li wrote:
> Hi,
>
> Recently, I am trying to run the data feed query on a cluster of 16 
> nodes. All the query run well without error. But each time after I 
> disconnect the feed, there is none of tweet data stored in dataverse. 
> However, when I try to run other AQL queries, the cluster can work 
> well. And I have ever used 4 nodes cluster to load Tweet data 
> successfully. I also checked the log file and find no error there. 
> This is wired. Does anyone know why? Has anyone ever used the data 
> feed function on a cluster of 16 nodes or more?
>
> Bests,
> Mingda