You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by mufy <mu...@gmail.com> on 2014/06/27 07:42:54 UTC
About Storage Plugin Instance Broadcast
Stated here
https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
that,
"When you add or update storage plugin instances on one node, Drill
broadcasts the information to all of the other Drill nodes so they have
identical storage plugin instances registered. You do not need to restart
any of the Drillbits when you add or update a storage plugin instance."
Does that mean if I were to have the following on one node:
[root@mufeed-drill1 ~]# cat
/opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
{
"storage": {
dfs: {
type: "file",
connection: "maprfs:///"
},
cp: {
type: "file",
connection: "classpath:///"
},
cp1: {
type: "file",
connection: "classpath:///"
},
}
}
The same content would get replicated to other nodes, on a file content
level? If yes, I do not see that happening. If not, please clarify.
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
Re: About Storage Plugin Instance Broadcast
Posted by Aditya <ad...@gmail.com>.
Yes, this is the only way today.
However, with completion of DRILL-77
<https://issues.apache.org/jira/browse/DRILL-77>, you can use REST APIs as
well.
On Sat, Jun 28, 2014 at 10:44 PM, mufy <mu...@gmail.com> wrote:
> I have seen that. Is that the only way of adding new storage plugins then?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
>
> On Sat, Jun 28, 2014 at 10:13 PM, Ramana Inukonda <ri...@maprtech.com>
> wrote:
>
> > Jason's comments are right for a clustered environment as well. Just
> > substitute localhost with any of the drillbit nodes in the cluster to
> > access the web UI.
> >
> > Regards
> > Ramana
> >
> > On Saturday, June 28, 2014, Jason Altekruse <al...@gmail.com>
> > wrote:
> >
> > > We have added a web server to Drill for showing query status
> information
> > as
> > > well as setting configuration options like storage plugins. To view the
> > > storage plugin management interface you can go to localhost:8047 if you
> > are
> > > running in local mode. If you are running from a cluster I believe you
> > > should be able to just substitute any of the IP addresses of the nodes,
> > but
> > > I haven't been working with Drill in a clustered environment, so
> someone
> > > else might have to chime in with more info.
> > >
> > > -Jason
> > >
> > >
> > > On Sat, Jun 28, 2014 at 7:12 AM, mufy <mufeed.usman@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > So as an end-user where exactly do I add my new storage plugin
> > > > configurations if I choose to add them in the future?
> > > >
> > > >
> > > > ---
> > > > Mufeed Usman
> > > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
> My
> > > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> LiveJournal
> > > > <http://mufeed.livejournal.com>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Jun 28, 2014 at 3:17 AM, Aditya <adityakishore@gmail.com
> > > <javascript:;>> wrote:
> > > >
> > > > > To be more precise, the configuration file "storage-plugins.json"
> has
> > > > been
> > > > > replaced with "bootstrap-storage-plugins.json" which resides inside
> > the
> > > > > Drill jars.
> > > > >
> > > > > When a cluster is started for the first time and no existing
> storage
> > > > plugin
> > > > > definition is found in the Zookeeper, this file will be read and
> > copied
> > > > > into the Zookeeper. From that point onward, the all Drillbits in
> the
> > > > > cluster will use only the information stored in the ZK and would
> > > ignore "
> > > > > bootstrap-storage-plugins.json" as well.
> > > > >
> > > > >
> > > > > On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com
> > > <javascript:;>> wrote:
> > > > >
> > > > > > The configuration file "storage-plugins.json" is no longer used
> by
> > > > Drill.
> > > > > >
> > > > > > Any change in the plugin configuration is stored only in the
> > > persistent
> > > > > > store (Zookeeper by default).
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com
> > > <javascript:;>> wrote:
> > > > > >
> > > > > >> Hello Anil,
> > > > > >>
> > > > > >> Hmm... so the ZK will have the info. Agreed. But what about the
> > > > > statement
> > > > > >> where it says that any new storage plugin added/updated will be
> > > > > broadcast
> > > > > >> to all the nodes running drillbit service. I took that statement
> > to
> > > > > mean -
> > > > > >> the config file will be synchronized across all the nodes
> running
> > > the
> > > > > >> drillbit service. Did I get it wrong?
> > > > > >>
> > > > > >>
> > > > > >> ---
> > > > > >> Mufeed Usman
> > > > > >> My LinkedIn <
> http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > |
> > > > My
> > > > > >> Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > LiveJournal
> > > > > >> <http://mufeed.livejournal.com>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <
> > > akumarb2010@gmail.com <javascript:;>>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Murfy,
> > > > > >> >
> > > > > >> > Based on my understanding, registered storage plugin info will
> > be
> > > > > >> stored on
> > > > > >> > ZK. As below.
> > > > > >> >
> > > > > >> > [zk: localhost:2181(CONNECTED) 10] ls
> /drill/sys.storage_plugins
> > > > > >> > [hive, cp, dfs, hbase]
> > > > > >> > [zk: localhost:2181(CONNECTED) 11] get
> > > > > /drill/sys.storage_plugins/hbase
> > > > > >> > {
> > > > > >> > "type" : "hbase",
> > > > > >> > "config" : {
> > > > > >> > "hbase.zookeeper.quorum" : "localhost",
> > > > > >> > "hbase.zookeeper.property.clientPort" : "2181"
> > > > > >> > },
> > > > > >> > "enabled" : true
> > > > > >> > }
> > > > > >> > cZxid = 0x428
> > > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > mZxid = 0x450
> > > > > >> > mtime = Tue Jun 24 13:31:41 IST 2014
> > > > > >> > pZxid = 0x428
> > > > > >> > cversion = 0
> > > > > >> > dataVersion = 1
> > > > > >> > aclVersion = 0
> > > > > >> > ephemeralOwner = 0x0
> > > > > >> > dataLength = 157
> > > > > >> > numChildren = 0
> > > > > >> > [zk: localhost:2181(CONNECTED) 12] get
> > > /drill/sys.storage_plugins/cp
> > > > > >> > {
> > > > > >> > "type" : "file",
> > > > > >> > "enabled" : true,
> > > > > >> > "connection" : "classpath:///",
> > > > > >> > "workspaces" : null,
> > > > > >> > "formats" : null
> > > > > >> > }
> > > > > >> > cZxid = 0x426
> > > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > mZxid = 0x426
> > > > > >> > mtime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > pZxid = 0x426
> > > > > >> > cversion = 0
> > > > > >> > dataVersion = 0
> > > > > >> > aclVersion = 0
> > > > > >> > ephemeralOwner = 0x0
> > > > > >> > dataLength = 118
> > > > > >> > numChildren = 0
> > > > > >> > [zk: localhost:2181(CONNECTED) 13]
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > B Anil Kumar.
> > > > > >> >
> > > > > >> > Thanks & Regards,
> > > > > >> > B Anil Kumar.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <
> mufeed.usman@gmail.com
> > > <javascript:;>>
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Stated here
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > > > > >> > > that,
> > > > > >> > >
> > > > > >> > > "When you add or update storage plugin instances on one
> node,
> > > > Drill
> > > > > >> > > broadcasts the information to all of the other Drill nodes
> so
> > > they
> > > > > >> have
> > > > > >> > > identical storage plugin instances registered. You do not
> need
> > > to
> > > > > >> restart
> > > > > >> > > any of the Drillbits when you add or update a storage plugin
> > > > > >> instance."
> > > > > >> > >
> > > > > >> > > Does that mean if I were to have the following on one node:
> > > > > >> > > [root@mufeed-drill1 ~]# cat
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > > > > >> > > {
> > > > > >> > > "storage": {
> > > > > >> > >
> > > > > >> > > dfs: {
> > > > > >> > > type: "file",
> > > > > >> > > connection: "maprfs:///"
> > > > > >> > > },
> > > > > >> > >
> > > > > >> > > cp: {
> > > > > >> > > type: "file",
> > > > > >> > > connection: "classpath:///"
> > > > > >> > > },
> > > > > >> > >
> > > > > >> > > cp1: {
> > > > > >> > > type: "file",
> > > > > >> > > connection: "classpath:///"
> > > > > >> > > },
> > > > > >> > > }
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > The same content would get replicated to other nodes, on a
> > file
> > > > > >> content
> > > > > >> > > level? If yes, I do not see that happening. If not, please
> > > > clarify.
> > > > > >> > >
> > > > > >> > > ---
> > > > > >> > > Mufeed Usman
> > > > > >> > > My LinkedIn <
> > > http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > > > |
> > > > > >> My
> > > > > >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > > > LiveJournal
> > > > > >> > > <http://mufeed.livejournal.com>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by mufy <mu...@gmail.com>.
I have seen that. Is that the only way of adding new storage plugins then?
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
On Sat, Jun 28, 2014 at 10:13 PM, Ramana Inukonda <ri...@maprtech.com>
wrote:
> Jason's comments are right for a clustered environment as well. Just
> substitute localhost with any of the drillbit nodes in the cluster to
> access the web UI.
>
> Regards
> Ramana
>
> On Saturday, June 28, 2014, Jason Altekruse <al...@gmail.com>
> wrote:
>
> > We have added a web server to Drill for showing query status information
> as
> > well as setting configuration options like storage plugins. To view the
> > storage plugin management interface you can go to localhost:8047 if you
> are
> > running in local mode. If you are running from a cluster I believe you
> > should be able to just substitute any of the IP addresses of the nodes,
> but
> > I haven't been working with Drill in a clustered environment, so someone
> > else might have to chime in with more info.
> >
> > -Jason
> >
> >
> > On Sat, Jun 28, 2014 at 7:12 AM, mufy <mufeed.usman@gmail.com
> > <javascript:;>> wrote:
> >
> > > So as an end-user where exactly do I add my new storage plugin
> > > configurations if I choose to add them in the future?
> > >
> > >
> > > ---
> > > Mufeed Usman
> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > > <http://mufeed.livejournal.com>
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Jun 28, 2014 at 3:17 AM, Aditya <adityakishore@gmail.com
> > <javascript:;>> wrote:
> > >
> > > > To be more precise, the configuration file "storage-plugins.json" has
> > > been
> > > > replaced with "bootstrap-storage-plugins.json" which resides inside
> the
> > > > Drill jars.
> > > >
> > > > When a cluster is started for the first time and no existing storage
> > > plugin
> > > > definition is found in the Zookeeper, this file will be read and
> copied
> > > > into the Zookeeper. From that point onward, the all Drillbits in the
> > > > cluster will use only the information stored in the ZK and would
> > ignore "
> > > > bootstrap-storage-plugins.json" as well.
> > > >
> > > >
> > > > On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > > The configuration file "storage-plugins.json" is no longer used by
> > > Drill.
> > > > >
> > > > > Any change in the plugin configuration is stored only in the
> > persistent
> > > > > store (Zookeeper by default).
> > > > >
> > > > >
> > > > > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com
> > <javascript:;>> wrote:
> > > > >
> > > > >> Hello Anil,
> > > > >>
> > > > >> Hmm... so the ZK will have the info. Agreed. But what about the
> > > > statement
> > > > >> where it says that any new storage plugin added/updated will be
> > > > broadcast
> > > > >> to all the nodes running drillbit service. I took that statement
> to
> > > > mean -
> > > > >> the config file will be synchronized across all the nodes running
> > the
> > > > >> drillbit service. Did I get it wrong?
> > > > >>
> > > > >>
> > > > >> ---
> > > > >> Mufeed Usman
> > > > >> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> |
> > > My
> > > > >> Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > LiveJournal
> > > > >> <http://mufeed.livejournal.com>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <
> > akumarb2010@gmail.com <javascript:;>>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Murfy,
> > > > >> >
> > > > >> > Based on my understanding, registered storage plugin info will
> be
> > > > >> stored on
> > > > >> > ZK. As below.
> > > > >> >
> > > > >> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> > > > >> > [hive, cp, dfs, hbase]
> > > > >> > [zk: localhost:2181(CONNECTED) 11] get
> > > > /drill/sys.storage_plugins/hbase
> > > > >> > {
> > > > >> > "type" : "hbase",
> > > > >> > "config" : {
> > > > >> > "hbase.zookeeper.quorum" : "localhost",
> > > > >> > "hbase.zookeeper.property.clientPort" : "2181"
> > > > >> > },
> > > > >> > "enabled" : true
> > > > >> > }
> > > > >> > cZxid = 0x428
> > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > >> > mZxid = 0x450
> > > > >> > mtime = Tue Jun 24 13:31:41 IST 2014
> > > > >> > pZxid = 0x428
> > > > >> > cversion = 0
> > > > >> > dataVersion = 1
> > > > >> > aclVersion = 0
> > > > >> > ephemeralOwner = 0x0
> > > > >> > dataLength = 157
> > > > >> > numChildren = 0
> > > > >> > [zk: localhost:2181(CONNECTED) 12] get
> > /drill/sys.storage_plugins/cp
> > > > >> > {
> > > > >> > "type" : "file",
> > > > >> > "enabled" : true,
> > > > >> > "connection" : "classpath:///",
> > > > >> > "workspaces" : null,
> > > > >> > "formats" : null
> > > > >> > }
> > > > >> > cZxid = 0x426
> > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > >> > mZxid = 0x426
> > > > >> > mtime = Tue Jun 24 13:30:18 IST 2014
> > > > >> > pZxid = 0x426
> > > > >> > cversion = 0
> > > > >> > dataVersion = 0
> > > > >> > aclVersion = 0
> > > > >> > ephemeralOwner = 0x0
> > > > >> > dataLength = 118
> > > > >> > numChildren = 0
> > > > >> > [zk: localhost:2181(CONNECTED) 13]
> > > > >> >
> > > > >> > Thanks,
> > > > >> > B Anil Kumar.
> > > > >> >
> > > > >> > Thanks & Regards,
> > > > >> > B Anil Kumar.
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mufeed.usman@gmail.com
> > <javascript:;>>
> > > > wrote:
> > > > >> >
> > > > >> > > Stated here
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > > > >> > > that,
> > > > >> > >
> > > > >> > > "When you add or update storage plugin instances on one node,
> > > Drill
> > > > >> > > broadcasts the information to all of the other Drill nodes so
> > they
> > > > >> have
> > > > >> > > identical storage plugin instances registered. You do not need
> > to
> > > > >> restart
> > > > >> > > any of the Drillbits when you add or update a storage plugin
> > > > >> instance."
> > > > >> > >
> > > > >> > > Does that mean if I were to have the following on one node:
> > > > >> > > [root@mufeed-drill1 ~]# cat
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > > > >> > > {
> > > > >> > > "storage": {
> > > > >> > >
> > > > >> > > dfs: {
> > > > >> > > type: "file",
> > > > >> > > connection: "maprfs:///"
> > > > >> > > },
> > > > >> > >
> > > > >> > > cp: {
> > > > >> > > type: "file",
> > > > >> > > connection: "classpath:///"
> > > > >> > > },
> > > > >> > >
> > > > >> > > cp1: {
> > > > >> > > type: "file",
> > > > >> > > connection: "classpath:///"
> > > > >> > > },
> > > > >> > > }
> > > > >> > > }
> > > > >> > >
> > > > >> > > The same content would get replicated to other nodes, on a
> file
> > > > >> content
> > > > >> > > level? If yes, I do not see that happening. If not, please
> > > clarify.
> > > > >> > >
> > > > >> > > ---
> > > > >> > > Mufeed Usman
> > > > >> > > My LinkedIn <
> > http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > > |
> > > > >> My
> > > > >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > > LiveJournal
> > > > >> > > <http://mufeed.livejournal.com>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by Ramana Inukonda <ri...@maprtech.com>.
Jason's comments are right for a clustered environment as well. Just
substitute localhost with any of the drillbit nodes in the cluster to
access the web UI.
Regards
Ramana
On Saturday, June 28, 2014, Jason Altekruse <al...@gmail.com>
wrote:
> We have added a web server to Drill for showing query status information as
> well as setting configuration options like storage plugins. To view the
> storage plugin management interface you can go to localhost:8047 if you are
> running in local mode. If you are running from a cluster I believe you
> should be able to just substitute any of the IP addresses of the nodes, but
> I haven't been working with Drill in a clustered environment, so someone
> else might have to chime in with more info.
>
> -Jason
>
>
> On Sat, Jun 28, 2014 at 7:12 AM, mufy <mufeed.usman@gmail.com
> <javascript:;>> wrote:
>
> > So as an end-user where exactly do I add my new storage plugin
> > configurations if I choose to add them in the future?
> >
> >
> > ---
> > Mufeed Usman
> > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > <http://mufeed.livejournal.com>
> >
> >
> >
> >
> >
> > On Sat, Jun 28, 2014 at 3:17 AM, Aditya <adityakishore@gmail.com
> <javascript:;>> wrote:
> >
> > > To be more precise, the configuration file "storage-plugins.json" has
> > been
> > > replaced with "bootstrap-storage-plugins.json" which resides inside the
> > > Drill jars.
> > >
> > > When a cluster is started for the first time and no existing storage
> > plugin
> > > definition is found in the Zookeeper, this file will be read and copied
> > > into the Zookeeper. From that point onward, the all Drillbits in the
> > > cluster will use only the information stored in the ZK and would
> ignore "
> > > bootstrap-storage-plugins.json" as well.
> > >
> > >
> > > On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com
> <javascript:;>> wrote:
> > >
> > > > The configuration file "storage-plugins.json" is no longer used by
> > Drill.
> > > >
> > > > Any change in the plugin configuration is stored only in the
> persistent
> > > > store (Zookeeper by default).
> > > >
> > > >
> > > > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com
> <javascript:;>> wrote:
> > > >
> > > >> Hello Anil,
> > > >>
> > > >> Hmm... so the ZK will have the info. Agreed. But what about the
> > > statement
> > > >> where it says that any new storage plugin added/updated will be
> > > broadcast
> > > >> to all the nodes running drillbit service. I took that statement to
> > > mean -
> > > >> the config file will be synchronized across all the nodes running
> the
> > > >> drillbit service. Did I get it wrong?
> > > >>
> > > >>
> > > >> ---
> > > >> Mufeed Usman
> > > >> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
> > My
> > > >> Social Cause <http://www.vision2016.org.in/> | My Blogs :
> LiveJournal
> > > >> <http://mufeed.livejournal.com>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <
> akumarb2010@gmail.com <javascript:;>>
> > > >> wrote:
> > > >>
> > > >> > Hi Murfy,
> > > >> >
> > > >> > Based on my understanding, registered storage plugin info will be
> > > >> stored on
> > > >> > ZK. As below.
> > > >> >
> > > >> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> > > >> > [hive, cp, dfs, hbase]
> > > >> > [zk: localhost:2181(CONNECTED) 11] get
> > > /drill/sys.storage_plugins/hbase
> > > >> > {
> > > >> > "type" : "hbase",
> > > >> > "config" : {
> > > >> > "hbase.zookeeper.quorum" : "localhost",
> > > >> > "hbase.zookeeper.property.clientPort" : "2181"
> > > >> > },
> > > >> > "enabled" : true
> > > >> > }
> > > >> > cZxid = 0x428
> > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > >> > mZxid = 0x450
> > > >> > mtime = Tue Jun 24 13:31:41 IST 2014
> > > >> > pZxid = 0x428
> > > >> > cversion = 0
> > > >> > dataVersion = 1
> > > >> > aclVersion = 0
> > > >> > ephemeralOwner = 0x0
> > > >> > dataLength = 157
> > > >> > numChildren = 0
> > > >> > [zk: localhost:2181(CONNECTED) 12] get
> /drill/sys.storage_plugins/cp
> > > >> > {
> > > >> > "type" : "file",
> > > >> > "enabled" : true,
> > > >> > "connection" : "classpath:///",
> > > >> > "workspaces" : null,
> > > >> > "formats" : null
> > > >> > }
> > > >> > cZxid = 0x426
> > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > >> > mZxid = 0x426
> > > >> > mtime = Tue Jun 24 13:30:18 IST 2014
> > > >> > pZxid = 0x426
> > > >> > cversion = 0
> > > >> > dataVersion = 0
> > > >> > aclVersion = 0
> > > >> > ephemeralOwner = 0x0
> > > >> > dataLength = 118
> > > >> > numChildren = 0
> > > >> > [zk: localhost:2181(CONNECTED) 13]
> > > >> >
> > > >> > Thanks,
> > > >> > B Anil Kumar.
> > > >> >
> > > >> > Thanks & Regards,
> > > >> > B Anil Kumar.
> > > >> >
> > > >> >
> > > >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mufeed.usman@gmail.com
> <javascript:;>>
> > > wrote:
> > > >> >
> > > >> > > Stated here
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > > >> > > that,
> > > >> > >
> > > >> > > "When you add or update storage plugin instances on one node,
> > Drill
> > > >> > > broadcasts the information to all of the other Drill nodes so
> they
> > > >> have
> > > >> > > identical storage plugin instances registered. You do not need
> to
> > > >> restart
> > > >> > > any of the Drillbits when you add or update a storage plugin
> > > >> instance."
> > > >> > >
> > > >> > > Does that mean if I were to have the following on one node:
> > > >> > > [root@mufeed-drill1 ~]# cat
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > > >> > > {
> > > >> > > "storage": {
> > > >> > >
> > > >> > > dfs: {
> > > >> > > type: "file",
> > > >> > > connection: "maprfs:///"
> > > >> > > },
> > > >> > >
> > > >> > > cp: {
> > > >> > > type: "file",
> > > >> > > connection: "classpath:///"
> > > >> > > },
> > > >> > >
> > > >> > > cp1: {
> > > >> > > type: "file",
> > > >> > > connection: "classpath:///"
> > > >> > > },
> > > >> > > }
> > > >> > > }
> > > >> > >
> > > >> > > The same content would get replicated to other nodes, on a file
> > > >> content
> > > >> > > level? If yes, I do not see that happening. If not, please
> > clarify.
> > > >> > >
> > > >> > > ---
> > > >> > > Mufeed Usman
> > > >> > > My LinkedIn <
> http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > |
> > > >> My
> > > >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > LiveJournal
> > > >> > > <http://mufeed.livejournal.com>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by Jason Altekruse <al...@gmail.com>.
We have added a web server to Drill for showing query status information as
well as setting configuration options like storage plugins. To view the
storage plugin management interface you can go to localhost:8047 if you are
running in local mode. If you are running from a cluster I believe you
should be able to just substitute any of the IP addresses of the nodes, but
I haven't been working with Drill in a clustered environment, so someone
else might have to chime in with more info.
-Jason
On Sat, Jun 28, 2014 at 7:12 AM, mufy <mu...@gmail.com> wrote:
> So as an end-user where exactly do I add my new storage plugin
> configurations if I choose to add them in the future?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
>
> On Sat, Jun 28, 2014 at 3:17 AM, Aditya <ad...@gmail.com> wrote:
>
> > To be more precise, the configuration file "storage-plugins.json" has
> been
> > replaced with "bootstrap-storage-plugins.json" which resides inside the
> > Drill jars.
> >
> > When a cluster is started for the first time and no existing storage
> plugin
> > definition is found in the Zookeeper, this file will be read and copied
> > into the Zookeeper. From that point onward, the all Drillbits in the
> > cluster will use only the information stored in the ZK and would ignore "
> > bootstrap-storage-plugins.json" as well.
> >
> >
> > On Fri, Jun 27, 2014 at 2:36 PM, Aditya <ad...@gmail.com> wrote:
> >
> > > The configuration file "storage-plugins.json" is no longer used by
> Drill.
> > >
> > > Any change in the plugin configuration is stored only in the persistent
> > > store (Zookeeper by default).
> > >
> > >
> > > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mu...@gmail.com> wrote:
> > >
> > >> Hello Anil,
> > >>
> > >> Hmm... so the ZK will have the info. Agreed. But what about the
> > statement
> > >> where it says that any new storage plugin added/updated will be
> > broadcast
> > >> to all the nodes running drillbit service. I took that statement to
> > mean -
> > >> the config file will be synchronized across all the nodes running the
> > >> drillbit service. Did I get it wrong?
> > >>
> > >>
> > >> ---
> > >> Mufeed Usman
> > >> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
> My
> > >> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > >> <http://mufeed.livejournal.com>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <ak...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Murfy,
> > >> >
> > >> > Based on my understanding, registered storage plugin info will be
> > >> stored on
> > >> > ZK. As below.
> > >> >
> > >> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> > >> > [hive, cp, dfs, hbase]
> > >> > [zk: localhost:2181(CONNECTED) 11] get
> > /drill/sys.storage_plugins/hbase
> > >> > {
> > >> > "type" : "hbase",
> > >> > "config" : {
> > >> > "hbase.zookeeper.quorum" : "localhost",
> > >> > "hbase.zookeeper.property.clientPort" : "2181"
> > >> > },
> > >> > "enabled" : true
> > >> > }
> > >> > cZxid = 0x428
> > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > >> > mZxid = 0x450
> > >> > mtime = Tue Jun 24 13:31:41 IST 2014
> > >> > pZxid = 0x428
> > >> > cversion = 0
> > >> > dataVersion = 1
> > >> > aclVersion = 0
> > >> > ephemeralOwner = 0x0
> > >> > dataLength = 157
> > >> > numChildren = 0
> > >> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
> > >> > {
> > >> > "type" : "file",
> > >> > "enabled" : true,
> > >> > "connection" : "classpath:///",
> > >> > "workspaces" : null,
> > >> > "formats" : null
> > >> > }
> > >> > cZxid = 0x426
> > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > >> > mZxid = 0x426
> > >> > mtime = Tue Jun 24 13:30:18 IST 2014
> > >> > pZxid = 0x426
> > >> > cversion = 0
> > >> > dataVersion = 0
> > >> > aclVersion = 0
> > >> > ephemeralOwner = 0x0
> > >> > dataLength = 118
> > >> > numChildren = 0
> > >> > [zk: localhost:2181(CONNECTED) 13]
> > >> >
> > >> > Thanks,
> > >> > B Anil Kumar.
> > >> >
> > >> > Thanks & Regards,
> > >> > B Anil Kumar.
> > >> >
> > >> >
> > >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com>
> > wrote:
> > >> >
> > >> > > Stated here
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > >> > > that,
> > >> > >
> > >> > > "When you add or update storage plugin instances on one node,
> Drill
> > >> > > broadcasts the information to all of the other Drill nodes so they
> > >> have
> > >> > > identical storage plugin instances registered. You do not need to
> > >> restart
> > >> > > any of the Drillbits when you add or update a storage plugin
> > >> instance."
> > >> > >
> > >> > > Does that mean if I were to have the following on one node:
> > >> > > [root@mufeed-drill1 ~]# cat
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > >> > > {
> > >> > > "storage": {
> > >> > >
> > >> > > dfs: {
> > >> > > type: "file",
> > >> > > connection: "maprfs:///"
> > >> > > },
> > >> > >
> > >> > > cp: {
> > >> > > type: "file",
> > >> > > connection: "classpath:///"
> > >> > > },
> > >> > >
> > >> > > cp1: {
> > >> > > type: "file",
> > >> > > connection: "classpath:///"
> > >> > > },
> > >> > > }
> > >> > > }
> > >> > >
> > >> > > The same content would get replicated to other nodes, on a file
> > >> content
> > >> > > level? If yes, I do not see that happening. If not, please
> clarify.
> > >> > >
> > >> > > ---
> > >> > > Mufeed Usman
> > >> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> |
> > >> My
> > >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > LiveJournal
> > >> > > <http://mufeed.livejournal.com>
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by mufy <mu...@gmail.com>.
So as an end-user where exactly do I add my new storage plugin
configurations if I choose to add them in the future?
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
On Sat, Jun 28, 2014 at 3:17 AM, Aditya <ad...@gmail.com> wrote:
> To be more precise, the configuration file "storage-plugins.json" has been
> replaced with "bootstrap-storage-plugins.json" which resides inside the
> Drill jars.
>
> When a cluster is started for the first time and no existing storage plugin
> definition is found in the Zookeeper, this file will be read and copied
> into the Zookeeper. From that point onward, the all Drillbits in the
> cluster will use only the information stored in the ZK and would ignore "
> bootstrap-storage-plugins.json" as well.
>
>
> On Fri, Jun 27, 2014 at 2:36 PM, Aditya <ad...@gmail.com> wrote:
>
> > The configuration file "storage-plugins.json" is no longer used by Drill.
> >
> > Any change in the plugin configuration is stored only in the persistent
> > store (Zookeeper by default).
> >
> >
> > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mu...@gmail.com> wrote:
> >
> >> Hello Anil,
> >>
> >> Hmm... so the ZK will have the info. Agreed. But what about the
> statement
> >> where it says that any new storage plugin added/updated will be
> broadcast
> >> to all the nodes running drillbit service. I took that statement to
> mean -
> >> the config file will be synchronized across all the nodes running the
> >> drillbit service. Did I get it wrong?
> >>
> >>
> >> ---
> >> Mufeed Usman
> >> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> >> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> >> <http://mufeed.livejournal.com>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <ak...@gmail.com>
> >> wrote:
> >>
> >> > Hi Murfy,
> >> >
> >> > Based on my understanding, registered storage plugin info will be
> >> stored on
> >> > ZK. As below.
> >> >
> >> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> >> > [hive, cp, dfs, hbase]
> >> > [zk: localhost:2181(CONNECTED) 11] get
> /drill/sys.storage_plugins/hbase
> >> > {
> >> > "type" : "hbase",
> >> > "config" : {
> >> > "hbase.zookeeper.quorum" : "localhost",
> >> > "hbase.zookeeper.property.clientPort" : "2181"
> >> > },
> >> > "enabled" : true
> >> > }
> >> > cZxid = 0x428
> >> > ctime = Tue Jun 24 13:30:18 IST 2014
> >> > mZxid = 0x450
> >> > mtime = Tue Jun 24 13:31:41 IST 2014
> >> > pZxid = 0x428
> >> > cversion = 0
> >> > dataVersion = 1
> >> > aclVersion = 0
> >> > ephemeralOwner = 0x0
> >> > dataLength = 157
> >> > numChildren = 0
> >> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
> >> > {
> >> > "type" : "file",
> >> > "enabled" : true,
> >> > "connection" : "classpath:///",
> >> > "workspaces" : null,
> >> > "formats" : null
> >> > }
> >> > cZxid = 0x426
> >> > ctime = Tue Jun 24 13:30:18 IST 2014
> >> > mZxid = 0x426
> >> > mtime = Tue Jun 24 13:30:18 IST 2014
> >> > pZxid = 0x426
> >> > cversion = 0
> >> > dataVersion = 0
> >> > aclVersion = 0
> >> > ephemeralOwner = 0x0
> >> > dataLength = 118
> >> > numChildren = 0
> >> > [zk: localhost:2181(CONNECTED) 13]
> >> >
> >> > Thanks,
> >> > B Anil Kumar.
> >> >
> >> > Thanks & Regards,
> >> > B Anil Kumar.
> >> >
> >> >
> >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com>
> wrote:
> >> >
> >> > > Stated here
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> >> > > that,
> >> > >
> >> > > "When you add or update storage plugin instances on one node, Drill
> >> > > broadcasts the information to all of the other Drill nodes so they
> >> have
> >> > > identical storage plugin instances registered. You do not need to
> >> restart
> >> > > any of the Drillbits when you add or update a storage plugin
> >> instance."
> >> > >
> >> > > Does that mean if I were to have the following on one node:
> >> > > [root@mufeed-drill1 ~]# cat
> >> > >
> >> > >
> >> >
> >>
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> >> > > {
> >> > > "storage": {
> >> > >
> >> > > dfs: {
> >> > > type: "file",
> >> > > connection: "maprfs:///"
> >> > > },
> >> > >
> >> > > cp: {
> >> > > type: "file",
> >> > > connection: "classpath:///"
> >> > > },
> >> > >
> >> > > cp1: {
> >> > > type: "file",
> >> > > connection: "classpath:///"
> >> > > },
> >> > > }
> >> > > }
> >> > >
> >> > > The same content would get replicated to other nodes, on a file
> >> content
> >> > > level? If yes, I do not see that happening. If not, please clarify.
> >> > >
> >> > > ---
> >> > > Mufeed Usman
> >> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
> >> My
> >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> LiveJournal
> >> > > <http://mufeed.livejournal.com>
> >> > >
> >> >
> >>
> >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by Aditya <ad...@gmail.com>.
To be more precise, the configuration file "storage-plugins.json" has been
replaced with "bootstrap-storage-plugins.json" which resides inside the
Drill jars.
When a cluster is started for the first time and no existing storage plugin
definition is found in the Zookeeper, this file will be read and copied
into the Zookeeper. From that point onward, the all Drillbits in the
cluster will use only the information stored in the ZK and would ignore "
bootstrap-storage-plugins.json" as well.
On Fri, Jun 27, 2014 at 2:36 PM, Aditya <ad...@gmail.com> wrote:
> The configuration file "storage-plugins.json" is no longer used by Drill.
>
> Any change in the plugin configuration is stored only in the persistent
> store (Zookeeper by default).
>
>
> On Fri, Jun 27, 2014 at 12:09 AM, mufy <mu...@gmail.com> wrote:
>
>> Hello Anil,
>>
>> Hmm... so the ZK will have the info. Agreed. But what about the statement
>> where it says that any new storage plugin added/updated will be broadcast
>> to all the nodes running drillbit service. I took that statement to mean -
>> the config file will be synchronized across all the nodes running the
>> drillbit service. Did I get it wrong?
>>
>>
>> ---
>> Mufeed Usman
>> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
>> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
>> <http://mufeed.livejournal.com>
>>
>>
>>
>>
>>
>> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <ak...@gmail.com>
>> wrote:
>>
>> > Hi Murfy,
>> >
>> > Based on my understanding, registered storage plugin info will be
>> stored on
>> > ZK. As below.
>> >
>> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
>> > [hive, cp, dfs, hbase]
>> > [zk: localhost:2181(CONNECTED) 11] get /drill/sys.storage_plugins/hbase
>> > {
>> > "type" : "hbase",
>> > "config" : {
>> > "hbase.zookeeper.quorum" : "localhost",
>> > "hbase.zookeeper.property.clientPort" : "2181"
>> > },
>> > "enabled" : true
>> > }
>> > cZxid = 0x428
>> > ctime = Tue Jun 24 13:30:18 IST 2014
>> > mZxid = 0x450
>> > mtime = Tue Jun 24 13:31:41 IST 2014
>> > pZxid = 0x428
>> > cversion = 0
>> > dataVersion = 1
>> > aclVersion = 0
>> > ephemeralOwner = 0x0
>> > dataLength = 157
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
>> > {
>> > "type" : "file",
>> > "enabled" : true,
>> > "connection" : "classpath:///",
>> > "workspaces" : null,
>> > "formats" : null
>> > }
>> > cZxid = 0x426
>> > ctime = Tue Jun 24 13:30:18 IST 2014
>> > mZxid = 0x426
>> > mtime = Tue Jun 24 13:30:18 IST 2014
>> > pZxid = 0x426
>> > cversion = 0
>> > dataVersion = 0
>> > aclVersion = 0
>> > ephemeralOwner = 0x0
>> > dataLength = 118
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 13]
>> >
>> > Thanks,
>> > B Anil Kumar.
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>> >
>> >
>> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com> wrote:
>> >
>> > > Stated here
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
>> > > that,
>> > >
>> > > "When you add or update storage plugin instances on one node, Drill
>> > > broadcasts the information to all of the other Drill nodes so they
>> have
>> > > identical storage plugin instances registered. You do not need to
>> restart
>> > > any of the Drillbits when you add or update a storage plugin
>> instance."
>> > >
>> > > Does that mean if I were to have the following on one node:
>> > > [root@mufeed-drill1 ~]# cat
>> > >
>> > >
>> >
>> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
>> > > {
>> > > "storage": {
>> > >
>> > > dfs: {
>> > > type: "file",
>> > > connection: "maprfs:///"
>> > > },
>> > >
>> > > cp: {
>> > > type: "file",
>> > > connection: "classpath:///"
>> > > },
>> > >
>> > > cp1: {
>> > > type: "file",
>> > > connection: "classpath:///"
>> > > },
>> > > }
>> > > }
>> > >
>> > > The same content would get replicated to other nodes, on a file
>> content
>> > > level? If yes, I do not see that happening. If not, please clarify.
>> > >
>> > > ---
>> > > Mufeed Usman
>> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
>> My
>> > > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
>> > > <http://mufeed.livejournal.com>
>> > >
>> >
>>
>
>
Re: About Storage Plugin Instance Broadcast
Posted by Aditya <ad...@gmail.com>.
The configuration file "storage-plugins.json" is no longer used by Drill.
Any change in the plugin configuration is stored only in the persistent
store (Zookeeper by default).
On Fri, Jun 27, 2014 at 12:09 AM, mufy <mu...@gmail.com> wrote:
> Hello Anil,
>
> Hmm... so the ZK will have the info. Agreed. But what about the statement
> where it says that any new storage plugin added/updated will be broadcast
> to all the nodes running drillbit service. I took that statement to mean -
> the config file will be synchronized across all the nodes running the
> drillbit service. Did I get it wrong?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
>
> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <ak...@gmail.com>
> wrote:
>
> > Hi Murfy,
> >
> > Based on my understanding, registered storage plugin info will be stored
> on
> > ZK. As below.
> >
> > [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> > [hive, cp, dfs, hbase]
> > [zk: localhost:2181(CONNECTED) 11] get /drill/sys.storage_plugins/hbase
> > {
> > "type" : "hbase",
> > "config" : {
> > "hbase.zookeeper.quorum" : "localhost",
> > "hbase.zookeeper.property.clientPort" : "2181"
> > },
> > "enabled" : true
> > }
> > cZxid = 0x428
> > ctime = Tue Jun 24 13:30:18 IST 2014
> > mZxid = 0x450
> > mtime = Tue Jun 24 13:31:41 IST 2014
> > pZxid = 0x428
> > cversion = 0
> > dataVersion = 1
> > aclVersion = 0
> > ephemeralOwner = 0x0
> > dataLength = 157
> > numChildren = 0
> > [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
> > {
> > "type" : "file",
> > "enabled" : true,
> > "connection" : "classpath:///",
> > "workspaces" : null,
> > "formats" : null
> > }
> > cZxid = 0x426
> > ctime = Tue Jun 24 13:30:18 IST 2014
> > mZxid = 0x426
> > mtime = Tue Jun 24 13:30:18 IST 2014
> > pZxid = 0x426
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x0
> > dataLength = 118
> > numChildren = 0
> > [zk: localhost:2181(CONNECTED) 13]
> >
> > Thanks,
> > B Anil Kumar.
> >
> > Thanks & Regards,
> > B Anil Kumar.
> >
> >
> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com> wrote:
> >
> > > Stated here
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > > that,
> > >
> > > "When you add or update storage plugin instances on one node, Drill
> > > broadcasts the information to all of the other Drill nodes so they have
> > > identical storage plugin instances registered. You do not need to
> restart
> > > any of the Drillbits when you add or update a storage plugin instance."
> > >
> > > Does that mean if I were to have the following on one node:
> > > [root@mufeed-drill1 ~]# cat
> > >
> > >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > > {
> > > "storage": {
> > >
> > > dfs: {
> > > type: "file",
> > > connection: "maprfs:///"
> > > },
> > >
> > > cp: {
> > > type: "file",
> > > connection: "classpath:///"
> > > },
> > >
> > > cp1: {
> > > type: "file",
> > > connection: "classpath:///"
> > > },
> > > }
> > > }
> > >
> > > The same content would get replicated to other nodes, on a file content
> > > level? If yes, I do not see that happening. If not, please clarify.
> > >
> > > ---
> > > Mufeed Usman
> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > > <http://mufeed.livejournal.com>
> > >
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by mufy <mu...@gmail.com>.
Hello Anil,
Hmm... so the ZK will have the info. Agreed. But what about the statement
where it says that any new storage plugin added/updated will be broadcast
to all the nodes running drillbit service. I took that statement to mean -
the config file will be synchronized across all the nodes running the
drillbit service. Did I get it wrong?
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <ak...@gmail.com> wrote:
> Hi Murfy,
>
> Based on my understanding, registered storage plugin info will be stored on
> ZK. As below.
>
> [zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
> [hive, cp, dfs, hbase]
> [zk: localhost:2181(CONNECTED) 11] get /drill/sys.storage_plugins/hbase
> {
> "type" : "hbase",
> "config" : {
> "hbase.zookeeper.quorum" : "localhost",
> "hbase.zookeeper.property.clientPort" : "2181"
> },
> "enabled" : true
> }
> cZxid = 0x428
> ctime = Tue Jun 24 13:30:18 IST 2014
> mZxid = 0x450
> mtime = Tue Jun 24 13:31:41 IST 2014
> pZxid = 0x428
> cversion = 0
> dataVersion = 1
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 157
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
> {
> "type" : "file",
> "enabled" : true,
> "connection" : "classpath:///",
> "workspaces" : null,
> "formats" : null
> }
> cZxid = 0x426
> ctime = Tue Jun 24 13:30:18 IST 2014
> mZxid = 0x426
> mtime = Tue Jun 24 13:30:18 IST 2014
> pZxid = 0x426
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 118
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 13]
>
> Thanks,
> B Anil Kumar.
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com> wrote:
>
> > Stated here
> >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > that,
> >
> > "When you add or update storage plugin instances on one node, Drill
> > broadcasts the information to all of the other Drill nodes so they have
> > identical storage plugin instances registered. You do not need to restart
> > any of the Drillbits when you add or update a storage plugin instance."
> >
> > Does that mean if I were to have the following on one node:
> > [root@mufeed-drill1 ~]# cat
> >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > {
> > "storage": {
> >
> > dfs: {
> > type: "file",
> > connection: "maprfs:///"
> > },
> >
> > cp: {
> > type: "file",
> > connection: "classpath:///"
> > },
> >
> > cp1: {
> > type: "file",
> > connection: "classpath:///"
> > },
> > }
> > }
> >
> > The same content would get replicated to other nodes, on a file content
> > level? If yes, I do not see that happening. If not, please clarify.
> >
> > ---
> > Mufeed Usman
> > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > <http://mufeed.livejournal.com>
> >
>
Re: About Storage Plugin Instance Broadcast
Posted by AnilKumar B <ak...@gmail.com>.
Hi Murfy,
Based on my understanding, registered storage plugin info will be stored on
ZK. As below.
[zk: localhost:2181(CONNECTED) 10] ls /drill/sys.storage_plugins
[hive, cp, dfs, hbase]
[zk: localhost:2181(CONNECTED) 11] get /drill/sys.storage_plugins/hbase
{
"type" : "hbase",
"config" : {
"hbase.zookeeper.quorum" : "localhost",
"hbase.zookeeper.property.clientPort" : "2181"
},
"enabled" : true
}
cZxid = 0x428
ctime = Tue Jun 24 13:30:18 IST 2014
mZxid = 0x450
mtime = Tue Jun 24 13:31:41 IST 2014
pZxid = 0x428
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 157
numChildren = 0
[zk: localhost:2181(CONNECTED) 12] get /drill/sys.storage_plugins/cp
{
"type" : "file",
"enabled" : true,
"connection" : "classpath:///",
"workspaces" : null,
"formats" : null
}
cZxid = 0x426
ctime = Tue Jun 24 13:30:18 IST 2014
mZxid = 0x426
mtime = Tue Jun 24 13:30:18 IST 2014
pZxid = 0x426
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 118
numChildren = 0
[zk: localhost:2181(CONNECTED) 13]
Thanks,
B Anil Kumar.
Thanks & Regards,
B Anil Kumar.
On Fri, Jun 27, 2014 at 11:12 AM, mufy <mu...@gmail.com> wrote:
> Stated here
>
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> that,
>
> "When you add or update storage plugin instances on one node, Drill
> broadcasts the information to all of the other Drill nodes so they have
> identical storage plugin instances registered. You do not need to restart
> any of the Drillbits when you add or update a storage plugin instance."
>
> Does that mean if I were to have the following on one node:
> [root@mufeed-drill1 ~]# cat
>
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> {
> "storage": {
>
> dfs: {
> type: "file",
> connection: "maprfs:///"
> },
>
> cp: {
> type: "file",
> connection: "classpath:///"
> },
>
> cp1: {
> type: "file",
> connection: "classpath:///"
> },
> }
> }
>
> The same content would get replicated to other nodes, on a file content
> level? If yes, I do not see that happening. If not, please clarify.
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>