You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by David Kincaid <ki...@gmail.com> on 2016/07/01 16:54:01 UTC
Re: AWS EMR bootstrap script to install and configure Drill
Thanks, Paul. This does look like a good place to start. Unfortunately, it
fails right off the bat due to the emr/common library not being available.
Not being a Ruby guy, I'm not sure where to go from here. Is there some
package that I can easily install to get that library?
- Dave
Here's the error for reference:
/usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
`require': cannot load such file -- emr/common (LoadError)
from
/usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
`require'
from ./setup-drill:39:in `<main>'
On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren <PM...@commercehub.com>
wrote:
> Dave,
>
> This is a little bit old/outdated but may be better than others you found:
>
> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill
>
> We haven’t gotten around to updating the overall Watershed project for EMR
> 4, which is managed differently, nor more recent versions of Drill. Mostly
> because it still meets our needs. But that has discouraged us from
> announcing the project, so I’d really like to get it done one of these days.
>
> Paul Mogren
>
>
> On 6/20/16, 10:29 AM, "David Kincaid" <ki...@gmail.com> wrote:
>
> I was wondering if anyone had an up to date bootstrap script that installs
> and configures Drill on an AWS EMR cluster. I'm looking for one that will
> work with the stock Amazon EMR version and not a MapR EMR cluster. I've had
> too much trouble in the past with the MapR EMR bootstrap scripts themselves
> failing. Nowadays I stick to the stock Amazon EMR. I've done some web
> searching, but all I can seem to find are pretty old, outdated or in some
> cases non-existent scripts. So I know people have tried it in the past. I'm
> very new to Drill, so having something to start with at least would be a
> huge help.
>
> Thanks,
>
> Dave
>
>
>
Re: AWS EMR bootstrap script to install and configure Drill
Posted by David Kincaid <ki...@gmail.com>.
Yes, very well could be. A lot of big changes with version 4. I'll see if I
can figure out where that library may have gone. They have added Zookeeper
as a supported application that EMR can install at startup, so at least we
should be able to eliminate the Zookeeper install from the bootstrap script.
On Sat, Jul 2, 2016 at 12:00 PM, Paul Mogren <PM...@commercehub.com>
wrote:
> Could be a EMR 3 vs 4 difference.
>
>
> > On Jul 2, 2016, at 8:51 AM, David Kincaid <ki...@gmail.com>
> wrote:
> >
> > I tried to run it from the shell (ssh'd in) on the master node of a newly
> > spun up EMR cluster which used the AWS EMR 4.x.
> >
> > On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren <PM...@commercehub.com>
> > wrote:
> >
> >> Did you by chance attempt to run it outside of EMR? AWS is known to
> >> provide a local install of EMR integration libraries on the nodes, but
> not
> >> publish them to OSS repositories.
> >>
> >> This was derived from the old
> >> https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and
> >> worked for us fairly recently.
> >>
> >>
> >>
> >>>> On Jul 1, 2016, at 12:54 PM, David Kincaid <ki...@gmail.com>
> >>> wrote:
> >>>
> >>> Thanks, Paul. This does look like a good place to start. Unfortunately,
> >> it
> >>> fails right off the bat due to the emr/common library not being
> >> available.
> >>> Not being a Ruby guy, I'm not sure where to go from here. Is there some
> >>> package that I can easily install to get that library?
> >>>
> >>> - Dave
> >>>
> >>> Here's the error for reference:
> >>>
> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> >>> `require': cannot load such file -- emr/common (LoadError)
> >>> from
> >>>
> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> >>> `require'
> >>> from ./setup-drill:39:in `<main>'
> >>>
> >>>
> >>> On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren <PM...@commercehub.com>
> >>> wrote:
> >>>
> >>>> Dave,
> >>>>
> >>>> This is a little bit old/outdated but may be better than others you
> >> found:
> >>
> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill
> >>>>
> >>>> We haven’t gotten around to updating the overall Watershed project for
> >> EMR
> >>>> 4, which is managed differently, nor more recent versions of Drill.
> >> Mostly
> >>>> because it still meets our needs. But that has discouraged us from
> >>>> announcing the project, so I’d really like to get it done one of these
> >> days.
> >>>>
> >>>> Paul Mogren
> >>>>
> >>>>
> >>>> On 6/20/16, 10:29 AM, "David Kincaid" <ki...@gmail.com> wrote:
> >>>>
> >>>> I was wondering if anyone had an up to date bootstrap script that
> >> installs
> >>>> and configures Drill on an AWS EMR cluster. I'm looking for one that
> >> will
> >>>> work with the stock Amazon EMR version and not a MapR EMR cluster.
> I've
> >> had
> >>>> too much trouble in the past with the MapR EMR bootstrap scripts
> >> themselves
> >>>> failing. Nowadays I stick to the stock Amazon EMR. I've done some web
> >>>> searching, but all I can seem to find are pretty old, outdated or in
> >> some
> >>>> cases non-existent scripts. So I know people have tried it in the
> past.
> >> I'm
> >>>> very new to Drill, so having something to start with at least would
> be a
> >>>> huge help.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Dave
> >>
>
Re: AWS EMR bootstrap script to install and configure Drill
Posted by Paul Mogren <PM...@commercehub.com>.
Could be a EMR 3 vs 4 difference.
> On Jul 2, 2016, at 8:51 AM, David Kincaid <ki...@gmail.com> wrote:
>
> I tried to run it from the shell (ssh'd in) on the master node of a newly
> spun up EMR cluster which used the AWS EMR 4.x.
>
> On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren <PM...@commercehub.com>
> wrote:
>
>> Did you by chance attempt to run it outside of EMR? AWS is known to
>> provide a local install of EMR integration libraries on the nodes, but not
>> publish them to OSS repositories.
>>
>> This was derived from the old
>> https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and
>> worked for us fairly recently.
>>
>>
>>
>>>> On Jul 1, 2016, at 12:54 PM, David Kincaid <ki...@gmail.com>
>>> wrote:
>>>
>>> Thanks, Paul. This does look like a good place to start. Unfortunately,
>> it
>>> fails right off the bat due to the emr/common library not being
>> available.
>>> Not being a Ruby guy, I'm not sure where to go from here. Is there some
>>> package that I can easily install to get that library?
>>>
>>> - Dave
>>>
>>> Here's the error for reference:
>>> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
>>> `require': cannot load such file -- emr/common (LoadError)
>>> from
>>> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
>>> `require'
>>> from ./setup-drill:39:in `<main>'
>>>
>>>
>>> On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren <PM...@commercehub.com>
>>> wrote:
>>>
>>>> Dave,
>>>>
>>>> This is a little bit old/outdated but may be better than others you
>> found:
>> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill
>>>>
>>>> We haven’t gotten around to updating the overall Watershed project for
>> EMR
>>>> 4, which is managed differently, nor more recent versions of Drill.
>> Mostly
>>>> because it still meets our needs. But that has discouraged us from
>>>> announcing the project, so I’d really like to get it done one of these
>> days.
>>>>
>>>> Paul Mogren
>>>>
>>>>
>>>> On 6/20/16, 10:29 AM, "David Kincaid" <ki...@gmail.com> wrote:
>>>>
>>>> I was wondering if anyone had an up to date bootstrap script that
>> installs
>>>> and configures Drill on an AWS EMR cluster. I'm looking for one that
>> will
>>>> work with the stock Amazon EMR version and not a MapR EMR cluster. I've
>> had
>>>> too much trouble in the past with the MapR EMR bootstrap scripts
>> themselves
>>>> failing. Nowadays I stick to the stock Amazon EMR. I've done some web
>>>> searching, but all I can seem to find are pretty old, outdated or in
>> some
>>>> cases non-existent scripts. So I know people have tried it in the past.
>> I'm
>>>> very new to Drill, so having something to start with at least would be a
>>>> huge help.
>>>>
>>>> Thanks,
>>>>
>>>> Dave
>>
Re: AWS EMR bootstrap script to install and configure Drill
Posted by David Kincaid <ki...@gmail.com>.
I tried to run it from the shell (ssh'd in) on the master node of a newly
spun up EMR cluster which used the AWS EMR 4.x.
On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren <PM...@commercehub.com>
wrote:
> Did you by chance attempt to run it outside of EMR? AWS is known to
> provide a local install of EMR integration libraries on the nodes, but not
> publish them to OSS repositories.
>
> This was derived from the old
> https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and
> worked for us fairly recently.
>
>
>
> > On Jul 1, 2016, at 12:54 PM, David Kincaid <ki...@gmail.com>
> wrote:
> >
> > Thanks, Paul. This does look like a good place to start. Unfortunately,
> it
> > fails right off the bat due to the emr/common library not being
> available.
> > Not being a Ruby guy, I'm not sure where to go from here. Is there some
> > package that I can easily install to get that library?
> >
> > - Dave
> >
> > Here's the error for reference:
> > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> > `require': cannot load such file -- emr/common (LoadError)
> > from
> > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> > `require'
> > from ./setup-drill:39:in `<main>'
> >
> >
> > On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren <PM...@commercehub.com>
> > wrote:
> >
> >> Dave,
> >>
> >> This is a little bit old/outdated but may be better than others you
> found:
> >>
> >>
> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill
> >>
> >> We haven’t gotten around to updating the overall Watershed project for
> EMR
> >> 4, which is managed differently, nor more recent versions of Drill.
> Mostly
> >> because it still meets our needs. But that has discouraged us from
> >> announcing the project, so I’d really like to get it done one of these
> days.
> >>
> >> Paul Mogren
> >>
> >>
> >> On 6/20/16, 10:29 AM, "David Kincaid" <ki...@gmail.com> wrote:
> >>
> >> I was wondering if anyone had an up to date bootstrap script that
> installs
> >> and configures Drill on an AWS EMR cluster. I'm looking for one that
> will
> >> work with the stock Amazon EMR version and not a MapR EMR cluster. I've
> had
> >> too much trouble in the past with the MapR EMR bootstrap scripts
> themselves
> >> failing. Nowadays I stick to the stock Amazon EMR. I've done some web
> >> searching, but all I can seem to find are pretty old, outdated or in
> some
> >> cases non-existent scripts. So I know people have tried it in the past.
> I'm
> >> very new to Drill, so having something to start with at least would be a
> >> huge help.
> >>
> >> Thanks,
> >>
> >> Dave
> >>
> >>
> >>
>
Re: AWS EMR bootstrap script to install and configure Drill
Posted by Paul Mogren <PM...@commercehub.com>.
Did you by chance attempt to run it outside of EMR? AWS is known to provide a local install of EMR integration libraries on the nodes, but not publish them to OSS repositories.
This was derived from the old https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and worked for us fairly recently.
> On Jul 1, 2016, at 12:54 PM, David Kincaid <ki...@gmail.com> wrote:
>
> Thanks, Paul. This does look like a good place to start. Unfortunately, it
> fails right off the bat due to the emr/common library not being available.
> Not being a Ruby guy, I'm not sure where to go from here. Is there some
> package that I can easily install to get that library?
>
> - Dave
>
> Here's the error for reference:
> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> `require': cannot load such file -- emr/common (LoadError)
> from
> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in
> `require'
> from ./setup-drill:39:in `<main>'
>
>
> On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren <PM...@commercehub.com>
> wrote:
>
>> Dave,
>>
>> This is a little bit old/outdated but may be better than others you found:
>>
>> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill
>>
>> We haven’t gotten around to updating the overall Watershed project for EMR
>> 4, which is managed differently, nor more recent versions of Drill. Mostly
>> because it still meets our needs. But that has discouraged us from
>> announcing the project, so I’d really like to get it done one of these days.
>>
>> Paul Mogren
>>
>>
>> On 6/20/16, 10:29 AM, "David Kincaid" <ki...@gmail.com> wrote:
>>
>> I was wondering if anyone had an up to date bootstrap script that installs
>> and configures Drill on an AWS EMR cluster. I'm looking for one that will
>> work with the stock Amazon EMR version and not a MapR EMR cluster. I've had
>> too much trouble in the past with the MapR EMR bootstrap scripts themselves
>> failing. Nowadays I stick to the stock Amazon EMR. I've done some web
>> searching, but all I can seem to find are pretty old, outdated or in some
>> cases non-existent scripts. So I know people have tried it in the past. I'm
>> very new to Drill, so having something to start with at least would be a
>> huge help.
>>
>> Thanks,
>>
>> Dave
>>
>>
>>