You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@asterixdb.apache.org by sc...@informatik.hu-berlin.de on 2015/10/20 14:40:46 UTC

unable to load external data

Hello,

I have done a cluster setup of AsterixDB on four nodes. Everyhing is
running fine and I want to load some data into the system to run sum
bigger examples. However I am unable to do so using the description at

https://asterixdb.ics.uci.edu/documentation/aql/externaldata.html

I created a dataverse, a datatype and a dataset as follows:

create dataverse tpch;

use dataverse tpch
create type LineitemType as closed {
      orderkey:int32,
      partkey: int32,
      suppkey: int32,
      linenumber: int32,
      quantity: double,
      extendedprice: double,
      discount: double,
      tax: double,
      returnflag: string,
      linestatus: string,
      shipdate: string,
      commitdate: string,
      receiptdate: string,
      shipinstruct: string,
      shipmode: string,
      comment: string}

create dataset lineitem(LineitemType) if not exists primary key orderkey,
linenumber

as described on the homepage linked above there are two ways to load data
from, using either a reachable HDFS or the localFS. I have a running HDFS
within the same network containing the data I want to access and tried to
reach it like this:

load dataset lineitem using hdfs
(("hdfs"="hdfs://192.168.127.11:50040"),
("path"="/user/schultzem/lineitem.tbl"),
("input-format"="text-input-format"),
("format"="delimited-text"),
("delimiter"="|"));

However I get an error message

Unable to create adapter org.apache.hadoop.ipc.RemoteException: Server IPC
version 9 cannot communicate with client version 3 [AlgebricksException]

all I found out about this was an old Issue from 2013 that recommends an
older version of hadoop, which is not an option for me.

https://code.google.com/p/asterixdb/issues/detail?id=521

Is this somehow fixable?

The other option to load data from the localFS also throws an error.

load dataset lineitem using localfs
(("path"="192.168.127.21:///home/schultzem/tpch/TPCH_data_10GB/lineitem.tbl"),
    ("format"="delimited-text"),
    ("delimiter"="|"));

leads to

No node controllers found at the address: 192.168.127.21 [AsterixException]

which is the same error as for 127.0.0.1.

On the linked documentation about external datasets it is assumed that
AsterixDB is used in local mode. Is this the problem why I cannot reach
the cluster nodes?

Did I make a mistake accessing the data? How can I load data into the
database?

Regards, Max

Re: AsterixDB - query status / system status

Posted by Yingyi Bu <bu...@gmail.com>.

The ones I know are in our regression tests:
https://github.com/apache/incubator-asterixdb/tree/master/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/

But they are not the original ones --- we changed the filter conditions in
some queries to make sure their result set is not empty.
I guess Pouria may have more accurate ones.

Best,
Yingyi


On Wed, Oct 28, 2015 at 9:53 AM, Michael Carey <mj...@ics.uci.edu> wrote:

> Pouria indeed has a full set of TPC-H queries in AQL....
> (Yingyi does too, I believe.)
>
> On 10/28/15 9:49 AM, Ian Maxon wrote:
>
> Hi Max,
>
> Let me respond inline...
>
> I am currently trying to rewrite some of the TPCH Queries to run them as
> examples on AsterixDB. So far I was able to run some of the queries using
> the web client, but others do not work yet. Unfortunatly sometimes I do
> not get any error messages or results.
>
> I agree, the WebUI leaves something to be desired, especially for
> queries that take a long time.
> There is one minor amelioration to this however, that I find useful,
> the Hyracks admin console.
> It's on port 8888 on the CC at /adminconsole . It will show what jobs
> are running and which
> NCs are still registered with the CC.
>
> As for rewriting the TPC-H queries, I'm pretty sure we have these (but
> the ones in the tests folder aren't, they're toned-down versions). So
> maybe you don't have to rewrite anything, hopefully?
> @ Pouria, do you happen to have them handy?
>
> In general though, for more serious use, the HTTP api may be a better
> option, as it is more amenable to scripting.
>
>
> As I cannot tell if the queries are still
> running, or if an error occured that is just not displayed, is there a
> possibility to monitor the state of the system/query execution?
>
> The adminconsole will show if there's a job still running, but not
> exceptional states that cause the job to hang.
> The exceptions should be in the CC log always. Where that is depends
> on your managix configuration (but it's always called cc.log).
>
> >From time to time the system even crashes and I cannot even shut it down
>
> using managix stop.
>
> Yes, unfortunately 'managix stop' just goes through the list of NC's
> and requests 'kill' on the process, not 'kill' and then 'kill -9'. In
> the cases where an NC fails to exit politely, it has to be done by
> hand for managix.
>
> By the way, what environment are you running everything on? Is it a cluster?
>
> Thanks,
> -Ian
>
> On Wed, Oct 28, 2015 at 9:22 AM,  <sc...@informatik.hu-berlin.de> <sc...@informatik.hu-berlin.de> wrote:
>
> Hello,
>
> by now I was able to successfully install AsterixDB from the latest master
> branch which also made it possible for me to load external data files into
> the system. Thanks a lot for the help!
>
> I am currently trying to rewrite some of the TPCH Queries to run them as
> examples on AsterixDB. So far I was able to run some of the queries using
> the web client, but others do not work yet. Unfortunatly sometimes I do
> not get any error messages or results. On execution the output area of the
> webclient turns blank as it usually does, but after some time I should get
> the results shown (or an error). As I cannot tell if the queries are still
> running, or if an error occured that is just not displayed, is there a
> possibility to monitor the state of the system/query execution?
>
> From time to time the system even crashes and I cannot even shut it down
> using managix stop.
>
> Regards, Max
>
>
>
> Your assumption is correct, for the latest AsterixDB master we usually
> depend on the development version of Hyracks
> (https://github.com/apache/incubator-asterixdb-hyracks/). 'mvn install
> -DskipTests' with Hyracks should do the trick.
>
> Thanks,
> -Ian
>
>
>
>

Re: AsterixDB - query status / system status

Posted by Michael Carey <mj...@ics.uci.edu>.

Pouria indeed has a full set of TPC-H queries in AQL....
(Yingyi does too, I believe.)

On 10/28/15 9:49 AM, Ian Maxon wrote:
> Hi Max,
>
> Let me respond inline...
>> I am currently trying to rewrite some of the TPCH Queries to run them as
>> examples on AsterixDB. So far I was able to run some of the queries using
>> the web client, but others do not work yet. Unfortunatly sometimes I do
>> not get any error messages or results.
> I agree, the WebUI leaves something to be desired, especially for
> queries that take a long time.
> There is one minor amelioration to this however, that I find useful,
> the Hyracks admin console.
> It's on port 8888 on the CC at /adminconsole . It will show what jobs
> are running and which
> NCs are still registered with the CC.
>
> As for rewriting the TPC-H queries, I'm pretty sure we have these (but
> the ones in the tests folder aren't, they're toned-down versions). So
> maybe you don't have to rewrite anything, hopefully?
> @ Pouria, do you happen to have them handy?
>
> In general though, for more serious use, the HTTP api may be a better
> option, as it is more amenable to scripting.
>
>> As I cannot tell if the queries are still
>> running, or if an error occured that is just not displayed, is there a
>> possibility to monitor the state of the system/query execution?
> The adminconsole will show if there's a job still running, but not
> exceptional states that cause the job to hang.
> The exceptions should be in the CC log always. Where that is depends
> on your managix configuration (but it's always called cc.log).
>
> >From time to time the system even crashes and I cannot even shut it down
>> using managix stop.
> Yes, unfortunately 'managix stop' just goes through the list of NC's
> and requests 'kill' on the process, not 'kill' and then 'kill -9'. In
> the cases where an NC fails to exit politely, it has to be done by
> hand for managix.
>
> By the way, what environment are you running everything on? Is it a cluster?
>
> Thanks,
> -Ian
>
> On Wed, Oct 28, 2015 at 9:22 AM,  <sc...@informatik.hu-berlin.de> wrote:
>> Hello,
>>
>> by now I was able to successfully install AsterixDB from the latest master
>> branch which also made it possible for me to load external data files into
>> the system. Thanks a lot for the help!
>>
>> I am currently trying to rewrite some of the TPCH Queries to run them as
>> examples on AsterixDB. So far I was able to run some of the queries using
>> the web client, but others do not work yet. Unfortunatly sometimes I do
>> not get any error messages or results. On execution the output area of the
>> webclient turns blank as it usually does, but after some time I should get
>> the results shown (or an error). As I cannot tell if the queries are still
>> running, or if an error occured that is just not displayed, is there a
>> possibility to monitor the state of the system/query execution?
>>
>>  From time to time the system even crashes and I cannot even shut it down
>> using managix stop.
>>
>> Regards, Max
>>
>>
>>> Your assumption is correct, for the latest AsterixDB master we usually
>>> depend on the development version of Hyracks
>>> (https://github.com/apache/incubator-asterixdb-hyracks/). 'mvn install
>>> -DskipTests' with Hyracks should do the trick.
>>>
>>> Thanks,
>>> -Ian
>>>

Re: AsterixDB - query status / system status

Posted by Pouria Pirzadeh <po...@gmail.com>.

Yingyi is correct about the ones under regression tests. Some of them are
modified to work fine on tiny scales of data.
You can find the actual TPCH queries, in AQL, under:

https://github.com/apache/incubator-asterixdb/tree/master/asterix-benchmarks/src/main/resources/tpc-h/queries

Let me know if you need more help about them.

Pouria

On Wed, Oct 28, 2015 at 10:34 AM, <sc...@informatik.hu-berlin.de> wrote:

> Hi Ian,
>
> thanks a lot for your help! Currently I prepare some of those tests on my
> local computer. Starting on Monday I will have access to 8 nodes of a
> cluster for another week.
>
> Regards, Max
>
> > Hi Max,
> >
> > Let me respond inline...
> >>I am currently trying to rewrite some of the TPCH Queries to run them as
> >>examples on AsterixDB. So far I was able to run some of the queries using
> >>the web client, but others do not work yet. Unfortunatly sometimes I do
> >>not get any error messages or results.
> >
> > I agree, the WebUI leaves something to be desired, especially for
> > queries that take a long time.
> > There is one minor amelioration to this however, that I find useful,
> > the Hyracks admin console.
> > It's on port 8888 on the CC at /adminconsole . It will show what jobs
> > are running and which
> > NCs are still registered with the CC.
> >
> > As for rewriting the TPC-H queries, I'm pretty sure we have these (but
> > the ones in the tests folder aren't, they're toned-down versions). So
> > maybe you don't have to rewrite anything, hopefully?
> > @ Pouria, do you happen to have them handy?
> >
> > In general though, for more serious use, the HTTP api may be a better
> > option, as it is more amenable to scripting.
> >
> >>As I cannot tell if the queries are still
> >>running, or if an error occured that is just not displayed, is there a
> >>possibility to monitor the state of the system/query execution?
> >
> > The adminconsole will show if there's a job still running, but not
> > exceptional states that cause the job to hang.
> > The exceptions should be in the CC log always. Where that is depends
> > on your managix configuration (but it's always called cc.log).
> >
> >>From time to time the system even crashes and I cannot even shut it down
> >>using managix stop.
> >
> > Yes, unfortunately 'managix stop' just goes through the list of NC's
> > and requests 'kill' on the process, not 'kill' and then 'kill -9'. In
> > the cases where an NC fails to exit politely, it has to be done by
> > hand for managix.
> >
> > By the way, what environment are you running everything on? Is it a
> > cluster?
> >
> > Thanks,
> > -Ian
> >
> > On Wed, Oct 28, 2015 at 9:22 AM,  <sc...@informatik.hu-berlin.de>
> > wrote:
> >> Hello,
> >>
> >> by now I was able to successfully install AsterixDB from the latest
> >> master
> >> branch which also made it possible for me to load external data files
> >> into
> >> the system. Thanks a lot for the help!
> >>
> >> I am currently trying to rewrite some of the TPCH Queries to run them as
> >> examples on AsterixDB. So far I was able to run some of the queries
> >> using
> >> the web client, but others do not work yet. Unfortunatly sometimes I do
> >> not get any error messages or results. On execution the output area of
> >> the
> >> webclient turns blank as it usually does, but after some time I should
> >> get
> >> the results shown (or an error). As I cannot tell if the queries are
> >> still
> >> running, or if an error occured that is just not displayed, is there a
> >> possibility to monitor the state of the system/query execution?
> >>
> >> From time to time the system even crashes and I cannot even shut it down
> >> using managix stop.
> >>
> >> Regards, Max
> >>
> >>
> >>> Your assumption is correct, for the latest AsterixDB master we usually
> >>> depend on the development version of Hyracks
> >>> (https://github.com/apache/incubator-asterixdb-hyracks/). 'mvn install
> >>> -DskipTests' with Hyracks should do the trick.
> >>>
> >>> Thanks,
> >>> -Ian
> >>>
> >>
> >
>
>
>