You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jacques Nadeau <ja...@apache.org> on 2014/06/11 21:36:18 UTC

Recent Commits - things to watch out for if you're upgrading

Hey Guys,

I've had a couple of questions about the commits that went in last night.
 Some things changed in configuration that people should be aware of.  They
are as follows:

- drill-override.conf is basically empty.  A sample drill-override is
available as well to see what some settings are that are available.  You
should move to this setup only migrate any settings that you have changed
from previous defaults.
- If you use your old settings, you'll likely to encounter a Zookeeper
exception.  This is because the drill.exec.zk.root no longer supports have
a leading slash (e.g. /drill).  To make it work, you need to either follow
the recommendation above or remove the leading slash.
- views, storage plugins and system settings are persisted across drillbit
restarts.
- view are persisted in writable workspaces (default one is dfs.tmp) as
json files called viewname.view.drill.
- Storage plugins and other drill settings are stored in a Drill PStores (a
persistent store for settings information).  In embedded mode they are
persisted to the local file system.  By default, in daemon/distributed
mode, they are stored in zookeeper.  In daemon/distributed mode, you can
also store them in HBase if you have that available as part of your cluster.
- In order to configure storage plugins, you need to use the web ui at
http://localhost:8047
- By default, Drill is initially populated with
bootstrap-storage-plugins.json in your classpath (Drill packages one in one
of the jars or you can put your own earlier in the classpath).  Once the
first node comes up and populates the storage plugin configuration, Drill
no longer uses or considers this file.


Let me know if there are other questions or issues that come up.  Also, be
sure to do a full clean build with this so you don't have any old/new file
conflicts.

thanks,
Jacques

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Aditya <ad...@gmail.com>.
That could happen if you have
"org.apache.drill.exec.store.sys.local.LocalPStoreProvider" set as the
class somewhere in the configuration.

As far as I searched, it is set only in "conf/drill-override-example.conf"
and should not be copied over to "conf/drill-override.conf"

However, there is a JIRA <https://issues.apache.org/jira/browse/DRILL-956>
to fix this just in case if some one want to use the local file store.


On Wed, Jun 11, 2014 at 4:54 PM, Xiao Meng <xi...@gmail.com> wrote:

>
> Oh, it is the default value of sys.store.provider.
> But without setting the value, I get the following exception when starting
> drillbit with distributed mode.
>
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:70)
>  at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:61)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:83)
> Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: A
> System Table provider was either not specified or could not be found or
> instantiated
> at
> org.apache.drill.exec.store.sys.PStoreRegistry.newPStoreProvider(PStoreRegistry.java:59)
>  at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:116)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:68)
>  ... 2 more
> Caused by: java.lang.NoSuchMethodException:
> org.apache.drill.exec.store.sys.local.LocalPStoreProvider.<init>(org.apache.drill.exec.store.sys.PStoreRegistry)
>  at java.lang.Class.getConstructor0(Class.java:2810)
> at java.lang.Class.getConstructor(Class.java:1718)
>  at
> org.apache.drill.exec.store.sys.PStoreRegistry.newPStoreProvider(PStoreRegistry.java:54)
> ... 4 more
> Exception in thread "ShutdownHook"
> java.lang.UnsupportedOperationException: Unknown handle type
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.unregister(ZKClusterCoordinator.java:151)
>  at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
> at
> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:187)
>
>
>
> On Wed, Jun 11, 2014 at 4:45 PM, Aditya <ad...@gmail.com> wrote:
>
>> "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider" is default
>> <
>> https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf#L98
>> >
>>
>> for distributed mode and you need not specify it explicitly.
>>
>> If you want to use HBase storage instead, you can specify "
>> org.apache.drill.exec.store.hbase.config.HBasePStoreProvider
>> <
>> https://github.com/apache/incubator-drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java
>> >"
>>
>> as the storage provider as follow
>>
>>   sys.store.provider: {
>> >     class:
>> "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
>> >     hbase: {
>> >       tableName : "drill_store",
>> >       config : {
>> >         "hbase.zookeeper.quorum" : "localhost",
>> >         "hbase.zookeeper.property.clientPort" : 2181
>> >       }
>> >     }
>> >   }
>> >
>>
>> The provider class is part of "drill-storage-hbase" project.
>>
>>
>>
>> On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I've tried it both in embedded mode and distributed mode and works fine
>> for
>> > me now.
>> >
>> > For the distributed mode, we need to set the sys.store.provider as
>> > zookeeper
>> >
>> > sys.store.provider: {
>> >  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
>> > },
>> >
>> > One question:
>> >
>> > > In daemon/distributed mode, you can also store them in HBase if you
>> have
>> > that available as part of your cluster.
>> > So do we need to write the plugin for HBase? I did not see any class in
>> > exec.store.sys.
>> >
>> >
>> > Hope it will help for the other people.
>> >
>> > BTW, can someone give the edit permission on the wiki? I can put this
>> into
>> > wiki. My user name is xiaom.
>> >
>> > Thanks,
>> >
>> > Xiao
>> >
>> >
>> > On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org>
>> > wrote:
>> >
>> > > Hey Guys,
>> > >
>> > > I've had a couple of questions about the commits that went in last
>> night.
>> > >  Some things changed in configuration that people should be aware of.
>> >  They
>> > > are as follows:
>> > >
>> > > - drill-override.conf is basically empty.  A sample drill-override is
>> > > available as well to see what some settings are that are available.
>>  You
>> > > should move to this setup only migrate any settings that you have
>> changed
>> > > from previous defaults.
>> > > - If you use your old settings, you'll likely to encounter a Zookeeper
>> > > exception.  This is because the drill.exec.zk.root no longer supports
>> > have
>> > > a leading slash (e.g. /drill).  To make it work, you need to either
>> > follow
>> > > the recommendation above or remove the leading slash.
>> > > - views, storage plugins and system settings are persisted across
>> > drillbit
>> > > restarts.
>> > > - view are persisted in writable workspaces (default one is dfs.tmp)
>> as
>> > > json files called viewname.view.drill.
>> > > - Storage plugins and other drill settings are stored in a Drill
>> PStores
>> > (a
>> > > persistent store for settings information).  In embedded mode they are
>> > > persisted to the local file system.  By default, in daemon/distributed
>> > > mode, they are stored in zookeeper.  In daemon/distributed mode, you
>> can
>> > > also store them in HBase if you have that available as part of your
>> > > cluster.
>> > > - In order to configure storage plugins, you need to use the web ui at
>> > > http://localhost:8047
>> > > - By default, Drill is initially populated with
>> > > bootstrap-storage-plugins.json in your classpath (Drill packages one
>> in
>> > one
>> > > of the jars or you can put your own earlier in the classpath).  Once
>> the
>> > > first node comes up and populates the storage plugin configuration,
>> Drill
>> > > no longer uses or considers this file.
>> > >
>> > >
>> > > Let me know if there are other questions or issues that come up.
>>  Also,
>> > be
>> > > sure to do a full clean build with this so you don't have any old/new
>> > file
>> > > conflicts.
>> > >
>> > > thanks,
>> > > Jacques
>> > >
>> >
>>
>
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Xiao Meng <xi...@gmail.com>.
Oh, it is the default value of sys.store.provider.
But without setting the value, I get the following exception when starting
drillbit with distributed mode.

Exception in thread "main"
org.apache.drill.exec.exception.DrillbitStartupException: Failure while
initializing values in Drillbit.
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:70)
 at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:61)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:83)
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: A
System Table provider was either not specified or could not be found or
instantiated
at
org.apache.drill.exec.store.sys.PStoreRegistry.newPStoreProvider(PStoreRegistry.java:59)
 at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:116)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:68)
 ... 2 more
Caused by: java.lang.NoSuchMethodException:
org.apache.drill.exec.store.sys.local.LocalPStoreProvider.<init>(org.apache.drill.exec.store.sys.PStoreRegistry)
 at java.lang.Class.getConstructor0(Class.java:2810)
at java.lang.Class.getConstructor(Class.java:1718)
 at
org.apache.drill.exec.store.sys.PStoreRegistry.newPStoreProvider(PStoreRegistry.java:54)
... 4 more
Exception in thread "ShutdownHook" java.lang.UnsupportedOperationException:
Unknown handle type
at
org.apache.drill.exec.coord.zk.ZKClusterCoordinator.unregister(ZKClusterCoordinator.java:151)
 at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
at
org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:187)



On Wed, Jun 11, 2014 at 4:45 PM, Aditya <ad...@gmail.com> wrote:

> "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider" is default
> <
> https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf#L98
> >
> for distributed mode and you need not specify it explicitly.
>
> If you want to use HBase storage instead, you can specify "
> org.apache.drill.exec.store.hbase.config.HBasePStoreProvider
> <
> https://github.com/apache/incubator-drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java
> >"
> as the storage provider as follow
>
>   sys.store.provider: {
> >     class:
> "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
> >     hbase: {
> >       tableName : "drill_store",
> >       config : {
> >         "hbase.zookeeper.quorum" : "localhost",
> >         "hbase.zookeeper.property.clientPort" : 2181
> >       }
> >     }
> >   }
> >
>
> The provider class is part of "drill-storage-hbase" project.
>
>
>
> On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:
>
> > Hi,
> >
> > I've tried it both in embedded mode and distributed mode and works fine
> for
> > me now.
> >
> > For the distributed mode, we need to set the sys.store.provider as
> > zookeeper
> >
> > sys.store.provider: {
> >  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
> > },
> >
> > One question:
> >
> > > In daemon/distributed mode, you can also store them in HBase if you
> have
> > that available as part of your cluster.
> > So do we need to write the plugin for HBase? I did not see any class in
> > exec.store.sys.
> >
> >
> > Hope it will help for the other people.
> >
> > BTW, can someone give the edit permission on the wiki? I can put this
> into
> > wiki. My user name is xiaom.
> >
> > Thanks,
> >
> > Xiao
> >
> >
> > On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org>
> > wrote:
> >
> > > Hey Guys,
> > >
> > > I've had a couple of questions about the commits that went in last
> night.
> > >  Some things changed in configuration that people should be aware of.
> >  They
> > > are as follows:
> > >
> > > - drill-override.conf is basically empty.  A sample drill-override is
> > > available as well to see what some settings are that are available.
>  You
> > > should move to this setup only migrate any settings that you have
> changed
> > > from previous defaults.
> > > - If you use your old settings, you'll likely to encounter a Zookeeper
> > > exception.  This is because the drill.exec.zk.root no longer supports
> > have
> > > a leading slash (e.g. /drill).  To make it work, you need to either
> > follow
> > > the recommendation above or remove the leading slash.
> > > - views, storage plugins and system settings are persisted across
> > drillbit
> > > restarts.
> > > - view are persisted in writable workspaces (default one is dfs.tmp) as
> > > json files called viewname.view.drill.
> > > - Storage plugins and other drill settings are stored in a Drill
> PStores
> > (a
> > > persistent store for settings information).  In embedded mode they are
> > > persisted to the local file system.  By default, in daemon/distributed
> > > mode, they are stored in zookeeper.  In daemon/distributed mode, you
> can
> > > also store them in HBase if you have that available as part of your
> > > cluster.
> > > - In order to configure storage plugins, you need to use the web ui at
> > > http://localhost:8047
> > > - By default, Drill is initially populated with
> > > bootstrap-storage-plugins.json in your classpath (Drill packages one in
> > one
> > > of the jars or you can put your own earlier in the classpath).  Once
> the
> > > first node comes up and populates the storage plugin configuration,
> Drill
> > > no longer uses or considers this file.
> > >
> > >
> > > Let me know if there are other questions or issues that come up.  Also,
> > be
> > > sure to do a full clean build with this so you don't have any old/new
> > file
> > > conflicts.
> > >
> > > thanks,
> > > Jacques
> > >
> >
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Aditya <ad...@gmail.com>.
"org.apache.drill.exec.store.sys.zk.ZkPStoreProvider" is default
<https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf#L98>
for distributed mode and you need not specify it explicitly.

If you want to use HBase storage instead, you can specify "
org.apache.drill.exec.store.hbase.config.HBasePStoreProvider
<https://github.com/apache/incubator-drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java>"
as the storage provider as follow

  sys.store.provider: {
>     class: "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
>     hbase: {
>       tableName : "drill_store",
>       config : {
>         "hbase.zookeeper.quorum" : "localhost",
>         "hbase.zookeeper.property.clientPort" : 2181
>       }
>     }
>   }
>

The provider class is part of "drill-storage-hbase" project.



On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:

> Hi,
>
> I've tried it both in embedded mode and distributed mode and works fine for
> me now.
>
> For the distributed mode, we need to set the sys.store.provider as
> zookeeper
>
> sys.store.provider: {
>  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
> },
>
> One question:
>
> > In daemon/distributed mode, you can also store them in HBase if you have
> that available as part of your cluster.
> So do we need to write the plugin for HBase? I did not see any class in
> exec.store.sys.
>
>
> Hope it will help for the other people.
>
> BTW, can someone give the edit permission on the wiki? I can put this into
> wiki. My user name is xiaom.
>
> Thanks,
>
> Xiao
>
>
> On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > Hey Guys,
> >
> > I've had a couple of questions about the commits that went in last night.
> >  Some things changed in configuration that people should be aware of.
>  They
> > are as follows:
> >
> > - drill-override.conf is basically empty.  A sample drill-override is
> > available as well to see what some settings are that are available.  You
> > should move to this setup only migrate any settings that you have changed
> > from previous defaults.
> > - If you use your old settings, you'll likely to encounter a Zookeeper
> > exception.  This is because the drill.exec.zk.root no longer supports
> have
> > a leading slash (e.g. /drill).  To make it work, you need to either
> follow
> > the recommendation above or remove the leading slash.
> > - views, storage plugins and system settings are persisted across
> drillbit
> > restarts.
> > - view are persisted in writable workspaces (default one is dfs.tmp) as
> > json files called viewname.view.drill.
> > - Storage plugins and other drill settings are stored in a Drill PStores
> (a
> > persistent store for settings information).  In embedded mode they are
> > persisted to the local file system.  By default, in daemon/distributed
> > mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> > also store them in HBase if you have that available as part of your
> > cluster.
> > - In order to configure storage plugins, you need to use the web ui at
> > http://localhost:8047
> > - By default, Drill is initially populated with
> > bootstrap-storage-plugins.json in your classpath (Drill packages one in
> one
> > of the jars or you can put your own earlier in the classpath).  Once the
> > first node comes up and populates the storage plugin configuration, Drill
> > no longer uses or considers this file.
> >
> >
> > Let me know if there are other questions or issues that come up.  Also,
> be
> > sure to do a full clean build with this so you don't have any old/new
> file
> > conflicts.
> >
> > thanks,
> > Jacques
> >
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Aditya <ad...@gmail.com>.
"org.apache.drill.exec.store.sys.zk.ZkPStoreProvider" is default
<https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf#L98>
for distributed mode and you need not specify it explicitly.

If you want to use HBase storage instead, you can specify "
org.apache.drill.exec.store.hbase.config.HBasePStoreProvider
<https://github.com/apache/incubator-drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePStoreProvider.java>"
as the storage provider as follow

  sys.store.provider: {
>     class: "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
>     hbase: {
>       tableName : "drill_store",
>       config : {
>         "hbase.zookeeper.quorum" : "localhost",
>         "hbase.zookeeper.property.clientPort" : 2181
>       }
>     }
>   }
>

The provider class is part of "drill-storage-hbase" project.



On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:

> Hi,
>
> I've tried it both in embedded mode and distributed mode and works fine for
> me now.
>
> For the distributed mode, we need to set the sys.store.provider as
> zookeeper
>
> sys.store.provider: {
>  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
> },
>
> One question:
>
> > In daemon/distributed mode, you can also store them in HBase if you have
> that available as part of your cluster.
> So do we need to write the plugin for HBase? I did not see any class in
> exec.store.sys.
>
>
> Hope it will help for the other people.
>
> BTW, can someone give the edit permission on the wiki? I can put this into
> wiki. My user name is xiaom.
>
> Thanks,
>
> Xiao
>
>
> On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > Hey Guys,
> >
> > I've had a couple of questions about the commits that went in last night.
> >  Some things changed in configuration that people should be aware of.
>  They
> > are as follows:
> >
> > - drill-override.conf is basically empty.  A sample drill-override is
> > available as well to see what some settings are that are available.  You
> > should move to this setup only migrate any settings that you have changed
> > from previous defaults.
> > - If you use your old settings, you'll likely to encounter a Zookeeper
> > exception.  This is because the drill.exec.zk.root no longer supports
> have
> > a leading slash (e.g. /drill).  To make it work, you need to either
> follow
> > the recommendation above or remove the leading slash.
> > - views, storage plugins and system settings are persisted across
> drillbit
> > restarts.
> > - view are persisted in writable workspaces (default one is dfs.tmp) as
> > json files called viewname.view.drill.
> > - Storage plugins and other drill settings are stored in a Drill PStores
> (a
> > persistent store for settings information).  In embedded mode they are
> > persisted to the local file system.  By default, in daemon/distributed
> > mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> > also store them in HBase if you have that available as part of your
> > cluster.
> > - In order to configure storage plugins, you need to use the web ui at
> > http://localhost:8047
> > - By default, Drill is initially populated with
> > bootstrap-storage-plugins.json in your classpath (Drill packages one in
> one
> > of the jars or you can put your own earlier in the classpath).  Once the
> > first node comes up and populates the storage plugin configuration, Drill
> > no longer uses or considers this file.
> >
> >
> > Let me know if there are other questions or issues that come up.  Also,
> be
> > sure to do a full clean build with this so you don't have any old/new
> file
> > conflicts.
> >
> > thanks,
> > Jacques
> >
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Xiao Meng <xi...@cs.sfu.ca>.
Thanks, Tim. I have the edit permission now.

Best,

Xiao


On Wed, Jun 11, 2014 at 4:33 PM, Timothy Chen <tn...@gmail.com> wrote:

> Hi Xiao,
>
> I just added you to the wiki.
>
> Please try it now.
>
> Tim
>
> On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:
> > Hi,
> >
> > I've tried it both in embedded mode and distributed mode and works fine
> for
> > me now.
> >
> > For the distributed mode, we need to set the sys.store.provider as
> zookeeper
> >
> > sys.store.provider: {
> >  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
> > },
> >
> > One question:
> >
> >> In daemon/distributed mode, you can also store them in HBase if you have
> > that available as part of your cluster.
> > So do we need to write the plugin for HBase? I did not see any class in
> > exec.store.sys.
> >
> >
> > Hope it will help for the other people.
> >
> > BTW, can someone give the edit permission on the wiki? I can put this
> into
> > wiki. My user name is xiaom.
> >
> > Thanks,
> >
> > Xiao
> >
> >
> > On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
> >
> >> Hey Guys,
> >>
> >> I've had a couple of questions about the commits that went in last
> night.
> >>  Some things changed in configuration that people should be aware of.
>  They
> >> are as follows:
> >>
> >> - drill-override.conf is basically empty.  A sample drill-override is
> >> available as well to see what some settings are that are available.  You
> >> should move to this setup only migrate any settings that you have
> changed
> >> from previous defaults.
> >> - If you use your old settings, you'll likely to encounter a Zookeeper
> >> exception.  This is because the drill.exec.zk.root no longer supports
> have
> >> a leading slash (e.g. /drill).  To make it work, you need to either
> follow
> >> the recommendation above or remove the leading slash.
> >> - views, storage plugins and system settings are persisted across
> drillbit
> >> restarts.
> >> - view are persisted in writable workspaces (default one is dfs.tmp) as
> >> json files called viewname.view.drill.
> >> - Storage plugins and other drill settings are stored in a Drill
> PStores (a
> >> persistent store for settings information).  In embedded mode they are
> >> persisted to the local file system.  By default, in daemon/distributed
> >> mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> >> also store them in HBase if you have that available as part of your
> >> cluster.
> >> - In order to configure storage plugins, you need to use the web ui at
> >> http://localhost:8047
> >> - By default, Drill is initially populated with
> >> bootstrap-storage-plugins.json in your classpath (Drill packages one in
> one
> >> of the jars or you can put your own earlier in the classpath).  Once the
> >> first node comes up and populates the storage plugin configuration,
> Drill
> >> no longer uses or considers this file.
> >>
> >>
> >> Let me know if there are other questions or issues that come up.  Also,
> be
> >> sure to do a full clean build with this so you don't have any old/new
> file
> >> conflicts.
> >>
> >> thanks,
> >> Jacques
> >>
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Timothy Chen <tn...@gmail.com>.
Hi Xiao,

I just added you to the wiki.

Please try it now.

Tim

On Wed, Jun 11, 2014 at 4:23 PM, Xiao Meng <xi...@gmail.com> wrote:
> Hi,
>
> I've tried it both in embedded mode and distributed mode and works fine for
> me now.
>
> For the distributed mode, we need to set the sys.store.provider as zookeeper
>
> sys.store.provider: {
>  class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
> },
>
> One question:
>
>> In daemon/distributed mode, you can also store them in HBase if you have
> that available as part of your cluster.
> So do we need to write the plugin for HBase? I did not see any class in
> exec.store.sys.
>
>
> Hope it will help for the other people.
>
> BTW, can someone give the edit permission on the wiki? I can put this into
> wiki. My user name is xiaom.
>
> Thanks,
>
> Xiao
>
>
> On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
>
>> Hey Guys,
>>
>> I've had a couple of questions about the commits that went in last night.
>>  Some things changed in configuration that people should be aware of.  They
>> are as follows:
>>
>> - drill-override.conf is basically empty.  A sample drill-override is
>> available as well to see what some settings are that are available.  You
>> should move to this setup only migrate any settings that you have changed
>> from previous defaults.
>> - If you use your old settings, you'll likely to encounter a Zookeeper
>> exception.  This is because the drill.exec.zk.root no longer supports have
>> a leading slash (e.g. /drill).  To make it work, you need to either follow
>> the recommendation above or remove the leading slash.
>> - views, storage plugins and system settings are persisted across drillbit
>> restarts.
>> - view are persisted in writable workspaces (default one is dfs.tmp) as
>> json files called viewname.view.drill.
>> - Storage plugins and other drill settings are stored in a Drill PStores (a
>> persistent store for settings information).  In embedded mode they are
>> persisted to the local file system.  By default, in daemon/distributed
>> mode, they are stored in zookeeper.  In daemon/distributed mode, you can
>> also store them in HBase if you have that available as part of your
>> cluster.
>> - In order to configure storage plugins, you need to use the web ui at
>> http://localhost:8047
>> - By default, Drill is initially populated with
>> bootstrap-storage-plugins.json in your classpath (Drill packages one in one
>> of the jars or you can put your own earlier in the classpath).  Once the
>> first node comes up and populates the storage plugin configuration, Drill
>> no longer uses or considers this file.
>>
>>
>> Let me know if there are other questions or issues that come up.  Also, be
>> sure to do a full clean build with this so you don't have any old/new file
>> conflicts.
>>
>> thanks,
>> Jacques
>>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Xiao Meng <xi...@gmail.com>.
Hi,

I've tried it both in embedded mode and distributed mode and works fine for
me now.

For the distributed mode, we need to set the sys.store.provider as zookeeper

sys.store.provider: {
 class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
},

One question:

> In daemon/distributed mode, you can also store them in HBase if you have
that available as part of your cluster.
So do we need to write the plugin for HBase? I did not see any class in
exec.store.sys.


Hope it will help for the other people.

BTW, can someone give the edit permission on the wiki? I can put this into
wiki. My user name is xiaom.

Thanks,

Xiao


On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:

> Hey Guys,
>
> I've had a couple of questions about the commits that went in last night.
>  Some things changed in configuration that people should be aware of.  They
> are as follows:
>
> - drill-override.conf is basically empty.  A sample drill-override is
> available as well to see what some settings are that are available.  You
> should move to this setup only migrate any settings that you have changed
> from previous defaults.
> - If you use your old settings, you'll likely to encounter a Zookeeper
> exception.  This is because the drill.exec.zk.root no longer supports have
> a leading slash (e.g. /drill).  To make it work, you need to either follow
> the recommendation above or remove the leading slash.
> - views, storage plugins and system settings are persisted across drillbit
> restarts.
> - view are persisted in writable workspaces (default one is dfs.tmp) as
> json files called viewname.view.drill.
> - Storage plugins and other drill settings are stored in a Drill PStores (a
> persistent store for settings information).  In embedded mode they are
> persisted to the local file system.  By default, in daemon/distributed
> mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> also store them in HBase if you have that available as part of your
> cluster.
> - In order to configure storage plugins, you need to use the web ui at
> http://localhost:8047
> - By default, Drill is initially populated with
> bootstrap-storage-plugins.json in your classpath (Drill packages one in one
> of the jars or you can put your own earlier in the classpath).  Once the
> first node comes up and populates the storage plugin configuration, Drill
> no longer uses or considers this file.
>
>
> Let me know if there are other questions or issues that come up.  Also, be
> sure to do a full clean build with this so you don't have any old/new file
> conflicts.
>
> thanks,
> Jacques
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Ted Dunning <te...@gmail.com>.
Tim,

These are really good points.


On Wed, Jun 11, 2014 at 3:26 PM, Timothy Chen <tn...@gmail.com> wrote:

> Sorry just to clarify more, this isn't about MapR and I think they've
> done a great job already to engage with the community (ie: with folks
> like me).
>
> Just hoping to bring some awareness around Drill project itself to see
> how we can improve a bit on some visibility of the project.
>
> I know we're very early in the project so there are lots of heads down
> progress going on.
>
> Tim
>
> On Wed, Jun 11, 2014 at 3:03 PM, Timothy Chen <tn...@gmail.com> wrote:
> > Ok sounds good.
> >
> > I was hoping to see the larger commits like view persistence, PStore
> > go through reviewboard first as lots of times there isn't even a
> > design discussion and just straight code that just goes into trunk.
> >
> > I think to foster a better community outside of MapR, Drill should be
> > more on the open about big changes and involve the community a bit
> > more.
> >
> > Tim
> >
> > On Wed, Jun 11, 2014 at 2:38 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
> >> Hey Tim,
> >>
> >> Great question.  I've been doing review of posted patches for a most of
> the
> >> features before merging.  We also internally have a substantial smoke
> test
> >> suite that we run before doing merges.  We're in the process of trying
> to
> >> clean it up and make it available externally so that others can use
> this as
> >> an additional check before merging.  My thought would be that minor
> fixes
> >> that have gone through unit test + smoke test validation should be
> >> mergeable directly by committers.  Larger commits should go through a
> >> review process.  I haven't been perfect at this in the last little bit
> so
> >> thanks for the reminder.
> >>
> >> thanks,
> >> Jacques
> >>
> >>
> >> On Wed, Jun 11, 2014 at 1:01 PM, Timothy Chen <tn...@gmail.com>
> wrote:
> >>
> >>> While I think all the changes are great, for some reason most of the
> >>> commits doesn't go through any reviewboard anymore, or even have any
> jira
> >>> too.
> >>>
> >>> What is our process going forward? Do all commuters just commit to
> trunk?
> >>>
> >>> Tim
> >>>
> >>>
> >>> > On Jun 11, 2014, at 12:36 PM, Jacques Nadeau <ja...@apache.org>
> wrote:
> >>> >
> >>> > Hey Guys,
> >>> >
> >>> > I've had a couple of questions about the commits that went in last
> night.
> >>> > Some things changed in configuration that people should be aware of.
> >>>  They
> >>> > are as follows:
> >>> >
> >>> > - drill-override.conf is basically empty.  A sample drill-override is
> >>> > available as well to see what some settings are that are available.
>  You
> >>> > should move to this setup only migrate any settings that you have
> changed
> >>> > from previous defaults.
> >>> > - If you use your old settings, you'll likely to encounter a
> Zookeeper
> >>> > exception.  This is because the drill.exec.zk.root no longer supports
> >>> have
> >>> > a leading slash (e.g. /drill).  To make it work, you need to either
> >>> follow
> >>> > the recommendation above or remove the leading slash.
> >>> > - views, storage plugins and system settings are persisted across
> >>> drillbit
> >>> > restarts.
> >>> > - view are persisted in writable workspaces (default one is dfs.tmp)
> as
> >>> > json files called viewname.view.drill.
> >>> > - Storage plugins and other drill settings are stored in a Drill
> PStores
> >>> (a
> >>> > persistent store for settings information).  In embedded mode they
> are
> >>> > persisted to the local file system.  By default, in
> daemon/distributed
> >>> > mode, they are stored in zookeeper.  In daemon/distributed mode, you
> can
> >>> > also store them in HBase if you have that available as part of your
> >>> cluster.
> >>> > - In order to configure storage plugins, you need to use the web ui
> at
> >>> > http://localhost:8047
> >>> > - By default, Drill is initially populated with
> >>> > bootstrap-storage-plugins.json in your classpath (Drill packages one
> in
> >>> one
> >>> > of the jars or you can put your own earlier in the classpath).  Once
> the
> >>> > first node comes up and populates the storage plugin configuration,
> Drill
> >>> > no longer uses or considers this file.
> >>> >
> >>> >
> >>> > Let me know if there are other questions or issues that come up.
>  Also,
> >>> be
> >>> > sure to do a full clean build with this so you don't have any old/new
> >>> file
> >>> > conflicts.
> >>> >
> >>> > thanks,
> >>> > Jacques
> >>>
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Timothy Chen <tn...@gmail.com>.
Sorry just to clarify more, this isn't about MapR and I think they've
done a great job already to engage with the community (ie: with folks
like me).

Just hoping to bring some awareness around Drill project itself to see
how we can improve a bit on some visibility of the project.

I know we're very early in the project so there are lots of heads down
progress going on.

Tim

On Wed, Jun 11, 2014 at 3:03 PM, Timothy Chen <tn...@gmail.com> wrote:
> Ok sounds good.
>
> I was hoping to see the larger commits like view persistence, PStore
> go through reviewboard first as lots of times there isn't even a
> design discussion and just straight code that just goes into trunk.
>
> I think to foster a better community outside of MapR, Drill should be
> more on the open about big changes and involve the community a bit
> more.
>
> Tim
>
> On Wed, Jun 11, 2014 at 2:38 PM, Jacques Nadeau <ja...@apache.org> wrote:
>> Hey Tim,
>>
>> Great question.  I've been doing review of posted patches for a most of the
>> features before merging.  We also internally have a substantial smoke test
>> suite that we run before doing merges.  We're in the process of trying to
>> clean it up and make it available externally so that others can use this as
>> an additional check before merging.  My thought would be that minor fixes
>> that have gone through unit test + smoke test validation should be
>> mergeable directly by committers.  Larger commits should go through a
>> review process.  I haven't been perfect at this in the last little bit so
>> thanks for the reminder.
>>
>> thanks,
>> Jacques
>>
>>
>> On Wed, Jun 11, 2014 at 1:01 PM, Timothy Chen <tn...@gmail.com> wrote:
>>
>>> While I think all the changes are great, for some reason most of the
>>> commits doesn't go through any reviewboard anymore, or even have any jira
>>> too.
>>>
>>> What is our process going forward? Do all commuters just commit to trunk?
>>>
>>> Tim
>>>
>>>
>>> > On Jun 11, 2014, at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
>>> >
>>> > Hey Guys,
>>> >
>>> > I've had a couple of questions about the commits that went in last night.
>>> > Some things changed in configuration that people should be aware of.
>>>  They
>>> > are as follows:
>>> >
>>> > - drill-override.conf is basically empty.  A sample drill-override is
>>> > available as well to see what some settings are that are available.  You
>>> > should move to this setup only migrate any settings that you have changed
>>> > from previous defaults.
>>> > - If you use your old settings, you'll likely to encounter a Zookeeper
>>> > exception.  This is because the drill.exec.zk.root no longer supports
>>> have
>>> > a leading slash (e.g. /drill).  To make it work, you need to either
>>> follow
>>> > the recommendation above or remove the leading slash.
>>> > - views, storage plugins and system settings are persisted across
>>> drillbit
>>> > restarts.
>>> > - view are persisted in writable workspaces (default one is dfs.tmp) as
>>> > json files called viewname.view.drill.
>>> > - Storage plugins and other drill settings are stored in a Drill PStores
>>> (a
>>> > persistent store for settings information).  In embedded mode they are
>>> > persisted to the local file system.  By default, in daemon/distributed
>>> > mode, they are stored in zookeeper.  In daemon/distributed mode, you can
>>> > also store them in HBase if you have that available as part of your
>>> cluster.
>>> > - In order to configure storage plugins, you need to use the web ui at
>>> > http://localhost:8047
>>> > - By default, Drill is initially populated with
>>> > bootstrap-storage-plugins.json in your classpath (Drill packages one in
>>> one
>>> > of the jars or you can put your own earlier in the classpath).  Once the
>>> > first node comes up and populates the storage plugin configuration, Drill
>>> > no longer uses or considers this file.
>>> >
>>> >
>>> > Let me know if there are other questions or issues that come up.  Also,
>>> be
>>> > sure to do a full clean build with this so you don't have any old/new
>>> file
>>> > conflicts.
>>> >
>>> > thanks,
>>> > Jacques
>>>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Timothy Chen <tn...@gmail.com>.
Ok sounds good.

I was hoping to see the larger commits like view persistence, PStore
go through reviewboard first as lots of times there isn't even a
design discussion and just straight code that just goes into trunk.

I think to foster a better community outside of MapR, Drill should be
more on the open about big changes and involve the community a bit
more.

Tim

On Wed, Jun 11, 2014 at 2:38 PM, Jacques Nadeau <ja...@apache.org> wrote:
> Hey Tim,
>
> Great question.  I've been doing review of posted patches for a most of the
> features before merging.  We also internally have a substantial smoke test
> suite that we run before doing merges.  We're in the process of trying to
> clean it up and make it available externally so that others can use this as
> an additional check before merging.  My thought would be that minor fixes
> that have gone through unit test + smoke test validation should be
> mergeable directly by committers.  Larger commits should go through a
> review process.  I haven't been perfect at this in the last little bit so
> thanks for the reminder.
>
> thanks,
> Jacques
>
>
> On Wed, Jun 11, 2014 at 1:01 PM, Timothy Chen <tn...@gmail.com> wrote:
>
>> While I think all the changes are great, for some reason most of the
>> commits doesn't go through any reviewboard anymore, or even have any jira
>> too.
>>
>> What is our process going forward? Do all commuters just commit to trunk?
>>
>> Tim
>>
>>
>> > On Jun 11, 2014, at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
>> >
>> > Hey Guys,
>> >
>> > I've had a couple of questions about the commits that went in last night.
>> > Some things changed in configuration that people should be aware of.
>>  They
>> > are as follows:
>> >
>> > - drill-override.conf is basically empty.  A sample drill-override is
>> > available as well to see what some settings are that are available.  You
>> > should move to this setup only migrate any settings that you have changed
>> > from previous defaults.
>> > - If you use your old settings, you'll likely to encounter a Zookeeper
>> > exception.  This is because the drill.exec.zk.root no longer supports
>> have
>> > a leading slash (e.g. /drill).  To make it work, you need to either
>> follow
>> > the recommendation above or remove the leading slash.
>> > - views, storage plugins and system settings are persisted across
>> drillbit
>> > restarts.
>> > - view are persisted in writable workspaces (default one is dfs.tmp) as
>> > json files called viewname.view.drill.
>> > - Storage plugins and other drill settings are stored in a Drill PStores
>> (a
>> > persistent store for settings information).  In embedded mode they are
>> > persisted to the local file system.  By default, in daemon/distributed
>> > mode, they are stored in zookeeper.  In daemon/distributed mode, you can
>> > also store them in HBase if you have that available as part of your
>> cluster.
>> > - In order to configure storage plugins, you need to use the web ui at
>> > http://localhost:8047
>> > - By default, Drill is initially populated with
>> > bootstrap-storage-plugins.json in your classpath (Drill packages one in
>> one
>> > of the jars or you can put your own earlier in the classpath).  Once the
>> > first node comes up and populates the storage plugin configuration, Drill
>> > no longer uses or considers this file.
>> >
>> >
>> > Let me know if there are other questions or issues that come up.  Also,
>> be
>> > sure to do a full clean build with this so you don't have any old/new
>> file
>> > conflicts.
>> >
>> > thanks,
>> > Jacques
>>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Jacques Nadeau <ja...@apache.org>.
Hey Tim,

Great question.  I've been doing review of posted patches for a most of the
features before merging.  We also internally have a substantial smoke test
suite that we run before doing merges.  We're in the process of trying to
clean it up and make it available externally so that others can use this as
an additional check before merging.  My thought would be that minor fixes
that have gone through unit test + smoke test validation should be
mergeable directly by committers.  Larger commits should go through a
review process.  I haven't been perfect at this in the last little bit so
thanks for the reminder.

thanks,
Jacques


On Wed, Jun 11, 2014 at 1:01 PM, Timothy Chen <tn...@gmail.com> wrote:

> While I think all the changes are great, for some reason most of the
> commits doesn't go through any reviewboard anymore, or even have any jira
> too.
>
> What is our process going forward? Do all commuters just commit to trunk?
>
> Tim
>
>
> > On Jun 11, 2014, at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
> >
> > Hey Guys,
> >
> > I've had a couple of questions about the commits that went in last night.
> > Some things changed in configuration that people should be aware of.
>  They
> > are as follows:
> >
> > - drill-override.conf is basically empty.  A sample drill-override is
> > available as well to see what some settings are that are available.  You
> > should move to this setup only migrate any settings that you have changed
> > from previous defaults.
> > - If you use your old settings, you'll likely to encounter a Zookeeper
> > exception.  This is because the drill.exec.zk.root no longer supports
> have
> > a leading slash (e.g. /drill).  To make it work, you need to either
> follow
> > the recommendation above or remove the leading slash.
> > - views, storage plugins and system settings are persisted across
> drillbit
> > restarts.
> > - view are persisted in writable workspaces (default one is dfs.tmp) as
> > json files called viewname.view.drill.
> > - Storage plugins and other drill settings are stored in a Drill PStores
> (a
> > persistent store for settings information).  In embedded mode they are
> > persisted to the local file system.  By default, in daemon/distributed
> > mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> > also store them in HBase if you have that available as part of your
> cluster.
> > - In order to configure storage plugins, you need to use the web ui at
> > http://localhost:8047
> > - By default, Drill is initially populated with
> > bootstrap-storage-plugins.json in your classpath (Drill packages one in
> one
> > of the jars or you can put your own earlier in the classpath).  Once the
> > first node comes up and populates the storage plugin configuration, Drill
> > no longer uses or considers this file.
> >
> >
> > Let me know if there are other questions or issues that come up.  Also,
> be
> > sure to do a full clean build with this so you don't have any old/new
> file
> > conflicts.
> >
> > thanks,
> > Jacques
>

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Timothy Chen <tn...@gmail.com>.
While I think all the changes are great, for some reason most of the commits doesn't go through any reviewboard anymore, or even have any jira too.

What is our process going forward? Do all commuters just commit to trunk?

Tim


> On Jun 11, 2014, at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
> 
> Hey Guys,
> 
> I've had a couple of questions about the commits that went in last night.
> Some things changed in configuration that people should be aware of.  They
> are as follows:
> 
> - drill-override.conf is basically empty.  A sample drill-override is
> available as well to see what some settings are that are available.  You
> should move to this setup only migrate any settings that you have changed
> from previous defaults.
> - If you use your old settings, you'll likely to encounter a Zookeeper
> exception.  This is because the drill.exec.zk.root no longer supports have
> a leading slash (e.g. /drill).  To make it work, you need to either follow
> the recommendation above or remove the leading slash.
> - views, storage plugins and system settings are persisted across drillbit
> restarts.
> - view are persisted in writable workspaces (default one is dfs.tmp) as
> json files called viewname.view.drill.
> - Storage plugins and other drill settings are stored in a Drill PStores (a
> persistent store for settings information).  In embedded mode they are
> persisted to the local file system.  By default, in daemon/distributed
> mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> also store them in HBase if you have that available as part of your cluster.
> - In order to configure storage plugins, you need to use the web ui at
> http://localhost:8047
> - By default, Drill is initially populated with
> bootstrap-storage-plugins.json in your classpath (Drill packages one in one
> of the jars or you can put your own earlier in the classpath).  Once the
> first node comes up and populates the storage plugin configuration, Drill
> no longer uses or considers this file.
> 
> 
> Let me know if there are other questions or issues that come up.  Also, be
> sure to do a full clean build with this so you don't have any old/new file
> conflicts.
> 
> thanks,
> Jacques

Re: Recent Commits - things to watch out for if you're upgrading

Posted by Xiao Meng <xi...@gmail.com>.
Hi,

I've tried it both in embedded mode and distributed mode and works fine for
me now.

For the distributed mode, we need to set the sys.store.provider as zookeeper

sys.store.provider: {
 class: "org.apache.drill.exec.store.sys.zk.ZkPStoreProvider"
},

One question:

> In daemon/distributed mode, you can also store them in HBase if you have
that available as part of your cluster.
So do we need to write the plugin for HBase? I did not see any class in
exec.store.sys.


Hope it will help for the other people.

BTW, can someone give the edit permission on the wiki? I can put this into
wiki. My user name is xiaom.

Thanks,

Xiao


On Wed, Jun 11, 2014 at 12:36 PM, Jacques Nadeau <ja...@apache.org> wrote:

> Hey Guys,
>
> I've had a couple of questions about the commits that went in last night.
>  Some things changed in configuration that people should be aware of.  They
> are as follows:
>
> - drill-override.conf is basically empty.  A sample drill-override is
> available as well to see what some settings are that are available.  You
> should move to this setup only migrate any settings that you have changed
> from previous defaults.
> - If you use your old settings, you'll likely to encounter a Zookeeper
> exception.  This is because the drill.exec.zk.root no longer supports have
> a leading slash (e.g. /drill).  To make it work, you need to either follow
> the recommendation above or remove the leading slash.
> - views, storage plugins and system settings are persisted across drillbit
> restarts.
> - view are persisted in writable workspaces (default one is dfs.tmp) as
> json files called viewname.view.drill.
> - Storage plugins and other drill settings are stored in a Drill PStores (a
> persistent store for settings information).  In embedded mode they are
> persisted to the local file system.  By default, in daemon/distributed
> mode, they are stored in zookeeper.  In daemon/distributed mode, you can
> also store them in HBase if you have that available as part of your
> cluster.
> - In order to configure storage plugins, you need to use the web ui at
> http://localhost:8047
> - By default, Drill is initially populated with
> bootstrap-storage-plugins.json in your classpath (Drill packages one in one
> of the jars or you can put your own earlier in the classpath).  Once the
> first node comes up and populates the storage plugin configuration, Drill
> no longer uses or considers this file.
>
>
> Let me know if there are other questions or issues that come up.  Also, be
> sure to do a full clean build with this so you don't have any old/new file
> conflicts.
>
> thanks,
> Jacques
>