You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Conrad Crampton <co...@SecData.com> on 2016/03/08 17:32:59 UTC
StoreInKiteDataset - clustered?
Hi,
Having looked at the tests for StoreInKiteDataset processor I can’t see why my attempt at using this is failing with a "failed to process due to java.lang.IllegalArgumentException: Missing Hive MetaStore connection URI; rolling back session: java.lang.IllegalArgumentException: Missing Hive MetaStore connection URI” exception. My uri is dataset:hive:prd/logs which I created using the Kite CLI on one of my nodes in cluster.
One part of the puzzle that I don’t understand is where this processor gets its knowledge of the metadata in order to store in the underlying storage structure of the Kite dataset – in this case; hive.
Do I have to somehow replicate the kite metastore (should one exist) one each node of my NiFi cluster (or hadoop cluster) so NiFi finds it or am I missing some dependencies (as suggested by an previous thread that had no apparent successful conclusion)?
Any pointers welcome.
Thanks
Conrad
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Re: StoreInKiteDataset - clustered?
Posted by Conrad Crampton <co...@SecData.com>.
Bryan,
Great spot – thanks for the pointer. This isn’t how it is described in their user docs though!
All working now.
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Tuesday, 8 March 2016 at 17:05
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: StoreInKiteDataset - clustered?
Conrad,
I really have no experience with Kite, but the javadocs [1] suggest something like this should work:
dataset:hive://[metastore-host]:[metastore-port]/[path]
-Bryan
[1] http://kitesdk.org/docs/0.11.0/apidocs/org/kitesdk/data/DatasetRepositories.html#open(java.lang.String)
On Tue, Mar 8, 2016 at 11:32 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Having looked at the tests for StoreInKiteDataset processor I can’t see why my attempt at using this is failing with a "failed to process due to java.lang.IllegalArgumentException: Missing Hive MetaStore connection URI; rolling back session: java.lang.IllegalArgumentException: Missing Hive MetaStore connection URI” exception. My uri is dataset:hive:prd/logs which I created using the Kite CLI on one of my nodes in cluster.
One part of the puzzle that I don’t understand is where this processor gets its knowledge of the metadata in order to store in the underlying storage structure of the Kite dataset – in this case; hive.
Do I have to somehow replicate the kite metastore (should one exist) one each node of my NiFi cluster (or hadoop cluster) so NiFi finds it or am I missing some dependencies (as suggested by an previous thread that had no apparent successful conclusion)?
Any pointers welcome.
Thanks
Conrad
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/9aY8QupNoMrGX2PQPOmvUsrLibhXE7+SH!4rXeSq4YH7CxnPuKWAKeHHH7Zw0zmi8nXiLCl81uhue2rhFpgg0Q==> to report this email as spam.
Re: StoreInKiteDataset - clustered?
Posted by Bryan Bende <bb...@gmail.com>.
Conrad,
I really have no experience with Kite, but the javadocs [1] suggest
something like this should work:
dataset:hive://[metastore-host]:[metastore-port]/[path]
-Bryan
[1]
http://kitesdk.org/docs/0.11.0/apidocs/org/kitesdk/data/DatasetRepositories.html#open(java.lang.String)
On Tue, Mar 8, 2016 at 11:32 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
> Having looked at the tests for StoreInKiteDataset processor I can’t see
> why my attempt at using this is failing with a "failed to process due to
> java.lang.IllegalArgumentException: Missing Hive MetaStore connection URI;
> rolling back session: java.lang.IllegalArgumentException: Missing Hive
> MetaStore connection URI” exception. My uri is dataset:hive:prd/logs which
> I created using the Kite CLI on one of my nodes in cluster.
> One part of the puzzle that I don’t understand is where this processor
> gets its knowledge of the metadata in order to store in the underlying
> storage structure of the Kite dataset – in this case; hive.
> Do I have to somehow replicate the kite metastore (should one exist) one
> each node of my NiFi cluster (or hadoop cluster) so NiFi finds it or am I
> missing some dependencies (as suggested by an previous thread that had no
> apparent successful conclusion)?
>
> Any pointers welcome.
> Thanks
> Conrad
>
>
>
> SecureData, combating cyber threats
>
> ------------------------------
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>