Re: About Storage Plugin Instance Broadcast

[Aditya <ad...@gmail.com>]

Yes, this is the only way today.

However, with completion of DRILL-77
<https://issues.apache.org/jira/browse/DRILL-77>, you can use REST APIs as
well.


On Sat, Jun 28, 2014 at 10:44 PM, mufy <mu...@gmail.com> wrote:

> I have seen that. Is that the only way of adding new storage plugins then?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
>
> On Sat, Jun 28, 2014 at 10:13 PM, Ramana Inukonda <ri...@maprtech.com>
> wrote:
>
> > Jason's comments are right for a clustered environment as well. Just
> > substitute localhost with any of the drillbit nodes in the cluster to
> > access the web UI.
> >
> > Regards
> > Ramana
> >
> > On Saturday, June 28, 2014, Jason Altekruse <al...@gmail.com>
> > wrote:
> >
> > > We have added a web server to Drill for showing query status
> information
> > as
> > > well as setting configuration options like storage plugins. To view the
> > > storage plugin management interface you can go to localhost:8047 if you
> > are
> > > running in local mode. If you are running from a cluster I believe you
> > > should be able to just substitute any of the IP addresses of the nodes,
> > but
> > > I haven't been working with Drill in a clustered environment, so
> someone
> > > else might have to chime in with more info.
> > >
> > > -Jason
> > >
> > >
> > > On Sat, Jun 28, 2014 at 7:12 AM, mufy <mufeed.usman@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > So as an end-user where exactly do I add my new storage plugin
> > > > configurations if I choose to add them in the future?
> > > >
> > > >
> > > > ---
> > > > Mufeed Usman
> > > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> |
> My
> > > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> LiveJournal
> > > > <http://mufeed.livejournal.com>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Jun 28, 2014 at 3:17 AM, Aditya <adityakishore@gmail.com
> > > <javascript:;>> wrote:
> > > >
> > > > > To be more precise, the configuration file "storage-plugins.json"
> has
> > > > been
> > > > > replaced with "bootstrap-storage-plugins.json" which resides inside
> > the
> > > > > Drill jars.
> > > > >
> > > > > When a cluster is started for the first time and no existing
> storage
> > > > plugin
> > > > > definition is found in the Zookeeper, this file will be read and
> > copied
> > > > > into the Zookeeper. From that point onward, the all Drillbits in
> the
> > > > > cluster will use only the information stored in the ZK and would
> > > ignore "
> > > > > bootstrap-storage-plugins.json" as well.
> > > > >
> > > > >
> > > > > On Fri, Jun 27, 2014 at 2:36 PM, Aditya <adityakishore@gmail.com
> > > <javascript:;>> wrote:
> > > > >
> > > > > > The configuration file "storage-plugins.json" is no longer used
> by
> > > > Drill.
> > > > > >
> > > > > > Any change in the plugin configuration is stored only in the
> > > persistent
> > > > > > store (Zookeeper by default).
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 27, 2014 at 12:09 AM, mufy <mufeed.usman@gmail.com
> > > <javascript:;>> wrote:
> > > > > >
> > > > > >> Hello Anil,
> > > > > >>
> > > > > >> Hmm... so the ZK will have the info. Agreed. But what about the
> > > > > statement
> > > > > >> where it says that any new storage plugin added/updated will be
> > > > > broadcast
> > > > > >> to all the nodes running drillbit service. I took that statement
> > to
> > > > > mean -
> > > > > >> the config file will be synchronized across all the nodes
> running
> > > the
> > > > > >> drillbit service. Did I get it wrong?
> > > > > >>
> > > > > >>
> > > > > >> ---
> > > > > >> Mufeed Usman
> > > > > >> My LinkedIn <
> http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > |
> > > > My
> > > > > >> Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > LiveJournal
> > > > > >> <http://mufeed.livejournal.com>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jun 27, 2014 at 11:57 AM, AnilKumar B <
> > > akumarb2010@gmail.com <javascript:;>>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Murfy,
> > > > > >> >
> > > > > >> > Based on my understanding, registered storage plugin info will
> > be
> > > > > >> stored on
> > > > > >> > ZK. As below.
> > > > > >> >
> > > > > >> > [zk: localhost:2181(CONNECTED) 10] ls
> /drill/sys.storage_plugins
> > > > > >> > [hive, cp, dfs, hbase]
> > > > > >> > [zk: localhost:2181(CONNECTED) 11] get
> > > > > /drill/sys.storage_plugins/hbase
> > > > > >> > {
> > > > > >> >   "type" : "hbase",
> > > > > >> >   "config" : {
> > > > > >> >     "hbase.zookeeper.quorum" : "localhost",
> > > > > >> >     "hbase.zookeeper.property.clientPort" : "2181"
> > > > > >> >   },
> > > > > >> >   "enabled" : true
> > > > > >> > }
> > > > > >> > cZxid = 0x428
> > > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > mZxid = 0x450
> > > > > >> > mtime = Tue Jun 24 13:31:41 IST 2014
> > > > > >> > pZxid = 0x428
> > > > > >> > cversion = 0
> > > > > >> > dataVersion = 1
> > > > > >> > aclVersion = 0
> > > > > >> > ephemeralOwner = 0x0
> > > > > >> > dataLength = 157
> > > > > >> > numChildren = 0
> > > > > >> > [zk: localhost:2181(CONNECTED) 12] get
> > > /drill/sys.storage_plugins/cp
> > > > > >> > {
> > > > > >> >   "type" : "file",
> > > > > >> >   "enabled" : true,
> > > > > >> >   "connection" : "classpath:///",
> > > > > >> >   "workspaces" : null,
> > > > > >> >   "formats" : null
> > > > > >> > }
> > > > > >> > cZxid = 0x426
> > > > > >> > ctime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > mZxid = 0x426
> > > > > >> > mtime = Tue Jun 24 13:30:18 IST 2014
> > > > > >> > pZxid = 0x426
> > > > > >> > cversion = 0
> > > > > >> > dataVersion = 0
> > > > > >> > aclVersion = 0
> > > > > >> > ephemeralOwner = 0x0
> > > > > >> > dataLength = 118
> > > > > >> > numChildren = 0
> > > > > >> > [zk: localhost:2181(CONNECTED) 13]
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > B Anil Kumar.
> > > > > >> >
> > > > > >> > Thanks & Regards,
> > > > > >> > B Anil Kumar.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Jun 27, 2014 at 11:12 AM, mufy <
> mufeed.usman@gmail.com
> > > <javascript:;>>
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Stated here
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/DRILL/Connecting+Apache+Drill+to+Data+Sources
> > > > > >> > > that,
> > > > > >> > >
> > > > > >> > > "When you add or update storage plugin instances on one
> node,
> > > > Drill
> > > > > >> > > broadcasts the information to all of the other Drill nodes
> so
> > > they
> > > > > >> have
> > > > > >> > > identical storage plugin instances registered. You do not
> need
> > > to
> > > > > >> restart
> > > > > >> > > any of the Drillbits when you add or update a storage plugin
> > > > > >> instance."
> > > > > >> > >
> > > > > >> > > Does that mean if I were to have the following on one node:
> > > > > >> > > [root@mufeed-drill1 ~]# cat
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> /opt/mapr/drill/drill-1.0.0/apache-drill-1.0.0-m2-incubating-SNAPSHOT/conf/storage-plugins.json
> > > > > >> > > {
> > > > > >> > >   "storage": {
> > > > > >> > >
> > > > > >> > >     dfs: {
> > > > > >> > >       type: "file",
> > > > > >> > >       connection: "maprfs:///"
> > > > > >> > >     },
> > > > > >> > >
> > > > > >> > >     cp: {
> > > > > >> > >       type: "file",
> > > > > >> > >       connection: "classpath:///"
> > > > > >> > >     },
> > > > > >> > >
> > > > > >> > >     cp1: {
> > > > > >> > >       type: "file",
> > > > > >> > >       connection: "classpath:///"
> > > > > >> > >     },
> > > > > >> > >       }
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > The same content would get replicated to other nodes, on a
> > file
> > > > > >> content
> > > > > >> > > level? If yes, I do not see that happening. If not, please
> > > > clarify.
> > > > > >> > >
> > > > > >> > > ---
> > > > > >> > > Mufeed Usman
> > > > > >> > > My LinkedIn <
> > > http://www.linkedin.com/pub/mufeed-usman/28/254/400>
> > > > |
> > > > > >> My
> > > > > >> > > Social Cause <http://www.vision2016.org.in/> | My Blogs :
> > > > > LiveJournal
> > > > > >> > > <http://mufeed.livejournal.com>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>