You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Odin Guillermo Caudillo Gallegos <od...@gmail.com> on 2016/05/13 15:48:16 UTC
workspaces
Is there a way to configure workspaces on a distributed installation?
Cause i only see the default plugin configuration but not the one that i
created.
Thanks
Re: workspaces
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
I believe Drill stores storage plugins in different places when running in
embedded mode vs distributed mode. Embedded mode uses local disk and
distributed mode uses Zookeeper.
On Fri, May 13, 2016 at 9:08 AM, Odin Guillermo Caudillo Gallegos <
odin.guillermo@gmail.com> wrote:
> The plugins are working fine in the embbed mode, but when i start the
> drillbit on each server and connect via drill-conf i don't see them.
> Do i need to configure another parameter apart from the zookeeper servers
> in the drill-override.conf file?
>
> 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <aengelbrecht@maprtech.com
> >:
>
> > If Drill was correctly installed in distributed mode the storage plugin
> > and workspaces will be used by the Drill cluster.
> >
> > Make sure the plugin and workspace was correctly configured and accepted.
> >
> > Are you using the WebUI or REST to configure the storage plugins?
> >
> > --Andries
> >
> > > On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
> > odin.guillermo@gmail.com> wrote:
> > >
> > > Is there a way to configure workspaces on a distributed installation?
> > > Cause i only see the default plugin configuration but not the one that
> i
> > > created.
> > >
> > > Thanks
> >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: workspaces
Posted by Andries Engelbrecht <ae...@maprtech.com>.
Alternatively you can just create the storage plugin json file, delete the old one and post the new one using the REST API.
See
https://drill.apache.org/docs/rest-api/#storage <https://drill.apache.org/docs/rest-api/#storage>
> On May 13, 2016, at 10:05 AM, Vince Gonzalez <vi...@gmail.com> wrote:
>
> I have used a pipeline involving jq and curl to modify storage plugins via
> the rest interface. The one below adds a workspace to the dfs plugin:
>
> curl -s localhost:8047/storage.json | jq '.[] | select(.name == "dfs")
> | .config.workspaces |= . + { "nypdmvc": { "location":
> "/Users/vince/data/nyc/nypdmvc", "writable": true,
> "defaultInputFormat": null} }' | curl -s -X POST -H "Content-Type:
> application/json" -d @- http://localhost:8047/storage/dfs.json <http://localhost:8047/storage/dfs.json>
>
> Note that this won't work as is if you have authentication enabled.
>
> On Friday, May 13, 2016, Odin Guillermo Caudillo Gallegos <
> odin.guillermo@gmail.com <ma...@gmail.com>> wrote:
>
>> I have the restriction to not configure it via web console, so is there a
>> way to configure them on the terminal?
>> Cause in embed mode, i only create the files on the /tmp/ directory via
>> terminal, also on the drill-override.conf file i use another path for the
>> plugins (with sys.store.provider.local.path)
>>
>> Thanks.
>>
>> 2016-05-13 11:33 GMT-05:00 Andries Engelbrecht <aengelbrecht@maprtech.com <ma...@maprtech.com>
>> <javascript:;>>:
>>
>>> You should start drill in distributed mode first and then configure the
>>> storage plugins.
>>> If you configure the storage plugins in embedded mode the information is
>>> stored in the tmp space instead of registered with ZK for the cluster to
>>> use.
>>>
>>> --Andries
>>>
>>>> On May 13, 2016, at 9:08 AM, Odin Guillermo Caudillo Gallegos <
>>> odin.guillermo@gmail.com <javascript:;>> wrote:
>>>>
>>>> The plugins are working fine in the embbed mode, but when i start the
>>>> drillbit on each server and connect via drill-conf i don't see them.
>>>> Do i need to configure another parameter apart from the zookeeper
>> servers
>>>> in the drill-override.conf file?
>>>>
>>>> 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <
>>> aengelbrecht@maprtech.com <javascript:;>>:
>>>>
>>>>> If Drill was correctly installed in distributed mode the storage
>> plugin
>>>>> and workspaces will be used by the Drill cluster.
>>>>>
>>>>> Make sure the plugin and workspace was correctly configured and
>>> accepted.
>>>>>
>>>>> Are you using the WebUI or REST to configure the storage plugins?
>>>>>
>>>>> --Andries
>>>>>
>>>>>> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
>>>>> odin.guillermo@gmail.com <javascript:;>> wrote:
>>>>>>
>>>>>> Is there a way to configure workspaces on a distributed installation?
>>>>>> Cause i only see the default plugin configuration but not the one
>> that
>>> i
>>>>>> created.
>>>>>>
>>>>>> Thanks
>>>>>
>>>>>
>>>
>>>
>>
>
>
> --
Re: workspaces
Posted by Odin Guillermo Caudillo Gallegos <od...@gmail.com>.
Ok i'll give it a try, thanks!
2016-05-13 12:05 GMT-05:00 Vince Gonzalez <vi...@gmail.com>:
> I have used a pipeline involving jq and curl to modify storage plugins via
> the rest interface. The one below adds a workspace to the dfs plugin:
>
> curl -s localhost:8047/storage.json | jq '.[] | select(.name == "dfs")
> | .config.workspaces |= . + { "nypdmvc": { "location":
> "/Users/vince/data/nyc/nypdmvc", "writable": true,
> "defaultInputFormat": null} }' | curl -s -X POST -H "Content-Type:
> application/json" -d @- http://localhost:8047/storage/dfs.json
>
> Note that this won't work as is if you have authentication enabled.
>
> On Friday, May 13, 2016, Odin Guillermo Caudillo Gallegos <
> odin.guillermo@gmail.com> wrote:
>
> > I have the restriction to not configure it via web console, so is there a
> > way to configure them on the terminal?
> > Cause in embed mode, i only create the files on the /tmp/ directory via
> > terminal, also on the drill-override.conf file i use another path for the
> > plugins (with sys.store.provider.local.path)
> >
> > Thanks.
> >
> > 2016-05-13 11:33 GMT-05:00 Andries Engelbrecht <
> aengelbrecht@maprtech.com
> > <javascript:;>>:
> >
> > > You should start drill in distributed mode first and then configure the
> > > storage plugins.
> > > If you configure the storage plugins in embedded mode the information
> is
> > > stored in the tmp space instead of registered with ZK for the cluster
> to
> > > use.
> > >
> > > --Andries
> > >
> > > > On May 13, 2016, at 9:08 AM, Odin Guillermo Caudillo Gallegos <
> > > odin.guillermo@gmail.com <javascript:;>> wrote:
> > > >
> > > > The plugins are working fine in the embbed mode, but when i start the
> > > > drillbit on each server and connect via drill-conf i don't see them.
> > > > Do i need to configure another parameter apart from the zookeeper
> > servers
> > > > in the drill-override.conf file?
> > > >
> > > > 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <
> > > aengelbrecht@maprtech.com <javascript:;>>:
> > > >
> > > >> If Drill was correctly installed in distributed mode the storage
> > plugin
> > > >> and workspaces will be used by the Drill cluster.
> > > >>
> > > >> Make sure the plugin and workspace was correctly configured and
> > > accepted.
> > > >>
> > > >> Are you using the WebUI or REST to configure the storage plugins?
> > > >>
> > > >> --Andries
> > > >>
> > > >>> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
> > > >> odin.guillermo@gmail.com <javascript:;>> wrote:
> > > >>>
> > > >>> Is there a way to configure workspaces on a distributed
> installation?
> > > >>> Cause i only see the default plugin configuration but not the one
> > that
> > > i
> > > >>> created.
> > > >>>
> > > >>> Thanks
> > > >>
> > > >>
> > >
> > >
> >
>
>
> --
>
Re: workspaces
Posted by Vince Gonzalez <vi...@gmail.com>.
I have used a pipeline involving jq and curl to modify storage plugins via
the rest interface. The one below adds a workspace to the dfs plugin:
curl -s localhost:8047/storage.json | jq '.[] | select(.name == "dfs")
| .config.workspaces |= . + { "nypdmvc": { "location":
"/Users/vince/data/nyc/nypdmvc", "writable": true,
"defaultInputFormat": null} }' | curl -s -X POST -H "Content-Type:
application/json" -d @- http://localhost:8047/storage/dfs.json
Note that this won't work as is if you have authentication enabled.
On Friday, May 13, 2016, Odin Guillermo Caudillo Gallegos <
odin.guillermo@gmail.com> wrote:
> I have the restriction to not configure it via web console, so is there a
> way to configure them on the terminal?
> Cause in embed mode, i only create the files on the /tmp/ directory via
> terminal, also on the drill-override.conf file i use another path for the
> plugins (with sys.store.provider.local.path)
>
> Thanks.
>
> 2016-05-13 11:33 GMT-05:00 Andries Engelbrecht <aengelbrecht@maprtech.com
> <javascript:;>>:
>
> > You should start drill in distributed mode first and then configure the
> > storage plugins.
> > If you configure the storage plugins in embedded mode the information is
> > stored in the tmp space instead of registered with ZK for the cluster to
> > use.
> >
> > --Andries
> >
> > > On May 13, 2016, at 9:08 AM, Odin Guillermo Caudillo Gallegos <
> > odin.guillermo@gmail.com <javascript:;>> wrote:
> > >
> > > The plugins are working fine in the embbed mode, but when i start the
> > > drillbit on each server and connect via drill-conf i don't see them.
> > > Do i need to configure another parameter apart from the zookeeper
> servers
> > > in the drill-override.conf file?
> > >
> > > 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <
> > aengelbrecht@maprtech.com <javascript:;>>:
> > >
> > >> If Drill was correctly installed in distributed mode the storage
> plugin
> > >> and workspaces will be used by the Drill cluster.
> > >>
> > >> Make sure the plugin and workspace was correctly configured and
> > accepted.
> > >>
> > >> Are you using the WebUI or REST to configure the storage plugins?
> > >>
> > >> --Andries
> > >>
> > >>> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
> > >> odin.guillermo@gmail.com <javascript:;>> wrote:
> > >>>
> > >>> Is there a way to configure workspaces on a distributed installation?
> > >>> Cause i only see the default plugin configuration but not the one
> that
> > i
> > >>> created.
> > >>>
> > >>> Thanks
> > >>
> > >>
> >
> >
>
--
Re: workspaces
Posted by Odin Guillermo Caudillo Gallegos <od...@gmail.com>.
I have the restriction to not configure it via web console, so is there a
way to configure them on the terminal?
Cause in embed mode, i only create the files on the /tmp/ directory via
terminal, also on the drill-override.conf file i use another path for the
plugins (with sys.store.provider.local.path)
Thanks.
2016-05-13 11:33 GMT-05:00 Andries Engelbrecht <ae...@maprtech.com>:
> You should start drill in distributed mode first and then configure the
> storage plugins.
> If you configure the storage plugins in embedded mode the information is
> stored in the tmp space instead of registered with ZK for the cluster to
> use.
>
> --Andries
>
> > On May 13, 2016, at 9:08 AM, Odin Guillermo Caudillo Gallegos <
> odin.guillermo@gmail.com> wrote:
> >
> > The plugins are working fine in the embbed mode, but when i start the
> > drillbit on each server and connect via drill-conf i don't see them.
> > Do i need to configure another parameter apart from the zookeeper servers
> > in the drill-override.conf file?
> >
> > 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <
> aengelbrecht@maprtech.com>:
> >
> >> If Drill was correctly installed in distributed mode the storage plugin
> >> and workspaces will be used by the Drill cluster.
> >>
> >> Make sure the plugin and workspace was correctly configured and
> accepted.
> >>
> >> Are you using the WebUI or REST to configure the storage plugins?
> >>
> >> --Andries
> >>
> >>> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
> >> odin.guillermo@gmail.com> wrote:
> >>>
> >>> Is there a way to configure workspaces on a distributed installation?
> >>> Cause i only see the default plugin configuration but not the one that
> i
> >>> created.
> >>>
> >>> Thanks
> >>
> >>
>
>
Re: workspaces
Posted by Andries Engelbrecht <ae...@maprtech.com>.
You should start drill in distributed mode first and then configure the storage plugins.
If you configure the storage plugins in embedded mode the information is stored in the tmp space instead of registered with ZK for the cluster to use.
--Andries
> On May 13, 2016, at 9:08 AM, Odin Guillermo Caudillo Gallegos <od...@gmail.com> wrote:
>
> The plugins are working fine in the embbed mode, but when i start the
> drillbit on each server and connect via drill-conf i don't see them.
> Do i need to configure another parameter apart from the zookeeper servers
> in the drill-override.conf file?
>
> 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <ae...@maprtech.com>:
>
>> If Drill was correctly installed in distributed mode the storage plugin
>> and workspaces will be used by the Drill cluster.
>>
>> Make sure the plugin and workspace was correctly configured and accepted.
>>
>> Are you using the WebUI or REST to configure the storage plugins?
>>
>> --Andries
>>
>>> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
>> odin.guillermo@gmail.com> wrote:
>>>
>>> Is there a way to configure workspaces on a distributed installation?
>>> Cause i only see the default plugin configuration but not the one that i
>>> created.
>>>
>>> Thanks
>>
>>
Re: workspaces
Posted by Odin Guillermo Caudillo Gallegos <od...@gmail.com>.
The plugins are working fine in the embbed mode, but when i start the
drillbit on each server and connect via drill-conf i don't see them.
Do i need to configure another parameter apart from the zookeeper servers
in the drill-override.conf file?
2016-05-13 11:01 GMT-05:00 Andries Engelbrecht <ae...@maprtech.com>:
> If Drill was correctly installed in distributed mode the storage plugin
> and workspaces will be used by the Drill cluster.
>
> Make sure the plugin and workspace was correctly configured and accepted.
>
> Are you using the WebUI or REST to configure the storage plugins?
>
> --Andries
>
> > On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <
> odin.guillermo@gmail.com> wrote:
> >
> > Is there a way to configure workspaces on a distributed installation?
> > Cause i only see the default plugin configuration but not the one that i
> > created.
> >
> > Thanks
>
>
Re: workspaces
Posted by Andries Engelbrecht <ae...@maprtech.com>.
If Drill was correctly installed in distributed mode the storage plugin and workspaces will be used by the Drill cluster.
Make sure the plugin and workspace was correctly configured and accepted.
Are you using the WebUI or REST to configure the storage plugins?
--Andries
> On May 13, 2016, at 8:48 AM, Odin Guillermo Caudillo Gallegos <od...@gmail.com> wrote:
>
> Is there a way to configure workspaces on a distributed installation?
> Cause i only see the default plugin configuration but not the one that i
> created.
>
> Thanks