You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@libcloud.apache.org by Marko Loparic <ma...@gmail.com> on 2014/06/24 23:33:11 UTC
list_images crashes (for ec2)
Hello,
The following script crashes when running on Amazon instance.
================================
from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver
EC2_ACCESS_ID = ...
EC2_SECRET = ...
Driver = get_driver(Provider.EC2_US_EAST)
conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
print "before"
conn.list_images()
print "after"
================================
The first print appears and "Killed" is displayed before the second print.
I suspect it is an error in etree module (apparently the list of
available images is huge).
I use list_images to get the python object of the desired image
image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
so that I can create a node with this image (create_node method)
Do you know if there is a way to create a node without using
list_images or if there is another way to avoid this crash?
I am using python 2.7.3 and libcloud 0.14.1.
Thanks a lot in advance!
Marko
Re: list_images crashes (for ec2)
Posted by Jayy Vis <ja...@pickle.io>.
You're welcome, Marko. Glad to help you.
On Thu, Jun 26, 2014 at 6:13 PM, Marko Loparic <ma...@gmail.com>
wrote:
> Hi Jayy,
>
> Sorry, I had misunderstood your initial message (I though NodeImage
> would give me a node but it gives me an image! duh...). I see now
> clearly what I have to do. I will make a test as soon as I solve my
> other issue (unrelated to libcloud).
>
> Thanks a lot for your help!
> Marko
>
> On Thu, Jun 26, 2014 at 12:59 PM, Jayy Vis <ja...@pickle.io> wrote:
> > You can launch as many instances as you want by specifying the count. And
> > the your create_node snippet looks fine.
> >
> > Are you facing any problem?
> >
> > BTW, create_node prepares request to amazon RunInstances api using the
> > arguments passed to it (image, size, ex_mincount, ex_maxcount). And the
> > image object being passed to create_node is libcloud's NodeImage type.
> >
> > image = kwargs["image"]
> > size = kwargs["size"]
> > params = {
> > 'Action': 'RunInstances',
> > 'ImageId': *image.id <http://image.id>*,
> > 'MinCount': str(kwargs.get('ex_mincount', '1')),
> > 'MaxCount': str(kwargs.get('ex_maxcount', '1')),
> > 'InstanceType': size.id
> > }
> >
> >
> > Best wishes,
> > Jayy Vis | @jayyvis
> > Co-founder, Pickle.io
> >
> >
> > On Thu, Jun 26, 2014 at 3:42 PM, Marko Loparic <ma...@gmail.com>
> > wrote:
> >
> >> Hello,
> >>
> >> Thanks a lot for all the answers! Each of them points to an
> >> interesting solution for my case. Yes, I was using a t1.micro instance
> >> indeed.
> >>
> >> Unfortunately I wasn't able to make tests yet (because of another
> >> totally independent firewall issue).
> >>
> >> Do you know if I can launch several nodes at once using what Jayy Vis
> >> proposed? I launch several machines at once and I suspect that
> >> launching one by one would take a lot of time. I use
> >>
> >> nodes = conn.create_node(name, image=image, size=size,
> >> ex_keyname=EC2_KEYNAME,
> >> ex_mincount=count, ex_maxcount=count)
> >>
> >> where count is usually between 500 and 2000. From my understanding of
> >> the code it seems that the create_node method for ec2
> >> (BaseEC2NodeDriver.create_method) uses an amazon RunInstances request
> >> and is not based on NodeImage.
> >>
> >> Thanks again,
> >> Marko
> >>
> >> On Wed, Jun 25, 2014 at 9:24 PM, Tomaz Muraus <to...@apache.org> wrote:
> >> > Yeah, what others have said.
> >> >
> >> > By default, calling list_images in the EC2 driver without any filters
> >> > returns a huge XML response. If lxml is not available (this feature is
> >> only
> >> > available in 0.15.0), parsing this response takes a while and
> consumes a
> >> > lot of memory.
> >> >
> >> > If you are not dynamically obtaining the image based on the name and
> you
> >> > already know the id, I would recommend you to use the approach Jayy
> Vis
> >> has
> >> > suggested.
> >> >
> >> > On Wed, Jun 25, 2014 at 7:13 AM, Jayy Vis <ja...@pickle.io> wrote:
> >> >
> >> >> create_node api of libcloud just uses image.id
> >> >>
> >> >> So you can simply create image object directly as follows.
> >> >>
> >> >> Driver = get_driver(Provider.EC2_US_EAST)
> >> >> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> >> >>
> >> >> from libcloud.compute.base import NodeImage
> >> >>
> >> >> ec2_ami_id='ami-123'
> >> >>
> >> >> image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id,
> extra=None)
> >> >>
> >> >>
> >> >>
> >> >> On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <
> >> LEHMANM@battelle.org>
> >> >> wrote:
> >> >>
> >> >> > Do you guys have lxml installed? With the dev release it will use
> lxml
> >> >> for
> >> >> > parsing if installed and that should help a bit.
> >> >> >
> >> >> >
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: Joseph Hall [perlhoser@gmail.com<mailto:perlhoser@gmail.com
> >]
> >> >> > Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
> >> >> > To: users@libcloud.apache.org
> >> >> > Subject: Re: list_images crashes (for ec2)
> >> >> >
> >> >> >
> >> >> > Where are you actually running this code from? I have found in
> recent
> >> >> > months that, when running libcloud against EC2 on a micro instance,
> >> >> > list_images() returns just too many images for the machine to
> handle,
> >> >> > and eventually the OOM killer destroys the process.
> >> >> >
> >> >> > On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <
> >> marko.loparic@gmail.com>
> >> >> > wrote:
> >> >> > > Hello,
> >> >> > >
> >> >> > > The following script crashes when running on Amazon instance.
> >> >> > >
> >> >> > > ================================
> >> >> > > from libcloud.compute.types import Provider
> >> >> > > from libcloud.compute.providers import get_driver
> >> >> > >
> >> >> > > EC2_ACCESS_ID = ...
> >> >> > > EC2_SECRET = ...
> >> >> > > Driver = get_driver(Provider.EC2_US_EAST)
> >> >> > > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> >> >> > >
> >> >> > > print "before"
> >> >> > > conn.list_images()
> >> >> > > print "after"
> >> >> > > ================================
> >> >> > >
> >> >> > > The first print appears and "Killed" is displayed before the
> second
> >> >> > print.
> >> >> > >
> >> >> > > I suspect it is an error in etree module (apparently the list of
> >> >> > > available images is huge).
> >> >> > >
> >> >> > > I use list_images to get the python object of the desired image
> >> >> > >
> >> >> > > image = [i for i in conn.list_images() if i.id ==
> EC2_AMI][0]
> >> >> > >
> >> >> > > so that I can create a node with this image (create_node method)
> >> >> > >
> >> >> > > Do you know if there is a way to create a node without using
> >> >> > > list_images or if there is another way to avoid this crash?
> >> >> > >
> >> >> > > I am using python 2.7.3 and libcloud 0.14.1.
> >> >> > >
> >> >> > > Thanks a lot in advance!
> >> >> > > Marko
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > "In order to create, you have to have the willingness, the desire
> to
> >> >> > be challenged, to be learning." -- Ferran Adria (speaking at
> Harvard,
> >> >> > 2011)
> >> >> >
> >> >>
> >>
>
Re: list_images crashes (for ec2)
Posted by Marko Loparic <ma...@gmail.com>.
Hi Jayy,
Sorry, I had misunderstood your initial message (I though NodeImage
would give me a node but it gives me an image! duh...). I see now
clearly what I have to do. I will make a test as soon as I solve my
other issue (unrelated to libcloud).
Thanks a lot for your help!
Marko
On Thu, Jun 26, 2014 at 12:59 PM, Jayy Vis <ja...@pickle.io> wrote:
> You can launch as many instances as you want by specifying the count. And
> the your create_node snippet looks fine.
>
> Are you facing any problem?
>
> BTW, create_node prepares request to amazon RunInstances api using the
> arguments passed to it (image, size, ex_mincount, ex_maxcount). And the
> image object being passed to create_node is libcloud's NodeImage type.
>
> image = kwargs["image"]
> size = kwargs["size"]
> params = {
> 'Action': 'RunInstances',
> 'ImageId': *image.id <http://image.id>*,
> 'MinCount': str(kwargs.get('ex_mincount', '1')),
> 'MaxCount': str(kwargs.get('ex_maxcount', '1')),
> 'InstanceType': size.id
> }
>
>
> Best wishes,
> Jayy Vis | @jayyvis
> Co-founder, Pickle.io
>
>
> On Thu, Jun 26, 2014 at 3:42 PM, Marko Loparic <ma...@gmail.com>
> wrote:
>
>> Hello,
>>
>> Thanks a lot for all the answers! Each of them points to an
>> interesting solution for my case. Yes, I was using a t1.micro instance
>> indeed.
>>
>> Unfortunately I wasn't able to make tests yet (because of another
>> totally independent firewall issue).
>>
>> Do you know if I can launch several nodes at once using what Jayy Vis
>> proposed? I launch several machines at once and I suspect that
>> launching one by one would take a lot of time. I use
>>
>> nodes = conn.create_node(name, image=image, size=size,
>> ex_keyname=EC2_KEYNAME,
>> ex_mincount=count, ex_maxcount=count)
>>
>> where count is usually between 500 and 2000. From my understanding of
>> the code it seems that the create_node method for ec2
>> (BaseEC2NodeDriver.create_method) uses an amazon RunInstances request
>> and is not based on NodeImage.
>>
>> Thanks again,
>> Marko
>>
>> On Wed, Jun 25, 2014 at 9:24 PM, Tomaz Muraus <to...@apache.org> wrote:
>> > Yeah, what others have said.
>> >
>> > By default, calling list_images in the EC2 driver without any filters
>> > returns a huge XML response. If lxml is not available (this feature is
>> only
>> > available in 0.15.0), parsing this response takes a while and consumes a
>> > lot of memory.
>> >
>> > If you are not dynamically obtaining the image based on the name and you
>> > already know the id, I would recommend you to use the approach Jayy Vis
>> has
>> > suggested.
>> >
>> > On Wed, Jun 25, 2014 at 7:13 AM, Jayy Vis <ja...@pickle.io> wrote:
>> >
>> >> create_node api of libcloud just uses image.id
>> >>
>> >> So you can simply create image object directly as follows.
>> >>
>> >> Driver = get_driver(Provider.EC2_US_EAST)
>> >> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>> >>
>> >> from libcloud.compute.base import NodeImage
>> >>
>> >> ec2_ami_id='ami-123'
>> >>
>> >> image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id, extra=None)
>> >>
>> >>
>> >>
>> >> On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <
>> LEHMANM@battelle.org>
>> >> wrote:
>> >>
>> >> > Do you guys have lxml installed? With the dev release it will use lxml
>> >> for
>> >> > parsing if installed and that should help a bit.
>> >> >
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
>> >> > Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
>> >> > To: users@libcloud.apache.org
>> >> > Subject: Re: list_images crashes (for ec2)
>> >> >
>> >> >
>> >> > Where are you actually running this code from? I have found in recent
>> >> > months that, when running libcloud against EC2 on a micro instance,
>> >> > list_images() returns just too many images for the machine to handle,
>> >> > and eventually the OOM killer destroys the process.
>> >> >
>> >> > On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <
>> marko.loparic@gmail.com>
>> >> > wrote:
>> >> > > Hello,
>> >> > >
>> >> > > The following script crashes when running on Amazon instance.
>> >> > >
>> >> > > ================================
>> >> > > from libcloud.compute.types import Provider
>> >> > > from libcloud.compute.providers import get_driver
>> >> > >
>> >> > > EC2_ACCESS_ID = ...
>> >> > > EC2_SECRET = ...
>> >> > > Driver = get_driver(Provider.EC2_US_EAST)
>> >> > > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>> >> > >
>> >> > > print "before"
>> >> > > conn.list_images()
>> >> > > print "after"
>> >> > > ================================
>> >> > >
>> >> > > The first print appears and "Killed" is displayed before the second
>> >> > print.
>> >> > >
>> >> > > I suspect it is an error in etree module (apparently the list of
>> >> > > available images is huge).
>> >> > >
>> >> > > I use list_images to get the python object of the desired image
>> >> > >
>> >> > > image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
>> >> > >
>> >> > > so that I can create a node with this image (create_node method)
>> >> > >
>> >> > > Do you know if there is a way to create a node without using
>> >> > > list_images or if there is another way to avoid this crash?
>> >> > >
>> >> > > I am using python 2.7.3 and libcloud 0.14.1.
>> >> > >
>> >> > > Thanks a lot in advance!
>> >> > > Marko
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > "In order to create, you have to have the willingness, the desire to
>> >> > be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
>> >> > 2011)
>> >> >
>> >>
>>
Re: list_images crashes (for ec2)
Posted by Jayy Vis <ja...@pickle.io>.
You can launch as many instances as you want by specifying the count. And
the your create_node snippet looks fine.
Are you facing any problem?
BTW, create_node prepares request to amazon RunInstances api using the
arguments passed to it (image, size, ex_mincount, ex_maxcount). And the
image object being passed to create_node is libcloud's NodeImage type.
image = kwargs["image"]
size = kwargs["size"]
params = {
'Action': 'RunInstances',
'ImageId': *image.id <http://image.id>*,
'MinCount': str(kwargs.get('ex_mincount', '1')),
'MaxCount': str(kwargs.get('ex_maxcount', '1')),
'InstanceType': size.id
}
Best wishes,
Jayy Vis | @jayyvis
Co-founder, Pickle.io
On Thu, Jun 26, 2014 at 3:42 PM, Marko Loparic <ma...@gmail.com>
wrote:
> Hello,
>
> Thanks a lot for all the answers! Each of them points to an
> interesting solution for my case. Yes, I was using a t1.micro instance
> indeed.
>
> Unfortunately I wasn't able to make tests yet (because of another
> totally independent firewall issue).
>
> Do you know if I can launch several nodes at once using what Jayy Vis
> proposed? I launch several machines at once and I suspect that
> launching one by one would take a lot of time. I use
>
> nodes = conn.create_node(name, image=image, size=size,
> ex_keyname=EC2_KEYNAME,
> ex_mincount=count, ex_maxcount=count)
>
> where count is usually between 500 and 2000. From my understanding of
> the code it seems that the create_node method for ec2
> (BaseEC2NodeDriver.create_method) uses an amazon RunInstances request
> and is not based on NodeImage.
>
> Thanks again,
> Marko
>
> On Wed, Jun 25, 2014 at 9:24 PM, Tomaz Muraus <to...@apache.org> wrote:
> > Yeah, what others have said.
> >
> > By default, calling list_images in the EC2 driver without any filters
> > returns a huge XML response. If lxml is not available (this feature is
> only
> > available in 0.15.0), parsing this response takes a while and consumes a
> > lot of memory.
> >
> > If you are not dynamically obtaining the image based on the name and you
> > already know the id, I would recommend you to use the approach Jayy Vis
> has
> > suggested.
> >
> > On Wed, Jun 25, 2014 at 7:13 AM, Jayy Vis <ja...@pickle.io> wrote:
> >
> >> create_node api of libcloud just uses image.id
> >>
> >> So you can simply create image object directly as follows.
> >>
> >> Driver = get_driver(Provider.EC2_US_EAST)
> >> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> >>
> >> from libcloud.compute.base import NodeImage
> >>
> >> ec2_ami_id='ami-123'
> >>
> >> image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id, extra=None)
> >>
> >>
> >>
> >> On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <
> LEHMANM@battelle.org>
> >> wrote:
> >>
> >> > Do you guys have lxml installed? With the dev release it will use lxml
> >> for
> >> > parsing if installed and that should help a bit.
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
> >> > Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
> >> > To: users@libcloud.apache.org
> >> > Subject: Re: list_images crashes (for ec2)
> >> >
> >> >
> >> > Where are you actually running this code from? I have found in recent
> >> > months that, when running libcloud against EC2 on a micro instance,
> >> > list_images() returns just too many images for the machine to handle,
> >> > and eventually the OOM killer destroys the process.
> >> >
> >> > On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <
> marko.loparic@gmail.com>
> >> > wrote:
> >> > > Hello,
> >> > >
> >> > > The following script crashes when running on Amazon instance.
> >> > >
> >> > > ================================
> >> > > from libcloud.compute.types import Provider
> >> > > from libcloud.compute.providers import get_driver
> >> > >
> >> > > EC2_ACCESS_ID = ...
> >> > > EC2_SECRET = ...
> >> > > Driver = get_driver(Provider.EC2_US_EAST)
> >> > > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> >> > >
> >> > > print "before"
> >> > > conn.list_images()
> >> > > print "after"
> >> > > ================================
> >> > >
> >> > > The first print appears and "Killed" is displayed before the second
> >> > print.
> >> > >
> >> > > I suspect it is an error in etree module (apparently the list of
> >> > > available images is huge).
> >> > >
> >> > > I use list_images to get the python object of the desired image
> >> > >
> >> > > image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
> >> > >
> >> > > so that I can create a node with this image (create_node method)
> >> > >
> >> > > Do you know if there is a way to create a node without using
> >> > > list_images or if there is another way to avoid this crash?
> >> > >
> >> > > I am using python 2.7.3 and libcloud 0.14.1.
> >> > >
> >> > > Thanks a lot in advance!
> >> > > Marko
> >> >
> >> >
> >> >
> >> > --
> >> > "In order to create, you have to have the willingness, the desire to
> >> > be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
> >> > 2011)
> >> >
> >>
>
Re: list_images crashes (for ec2)
Posted by Marko Loparic <ma...@gmail.com>.
Hello,
Thanks a lot for all the answers! Each of them points to an
interesting solution for my case. Yes, I was using a t1.micro instance
indeed.
Unfortunately I wasn't able to make tests yet (because of another
totally independent firewall issue).
Do you know if I can launch several nodes at once using what Jayy Vis
proposed? I launch several machines at once and I suspect that
launching one by one would take a lot of time. I use
nodes = conn.create_node(name, image=image, size=size,
ex_keyname=EC2_KEYNAME,
ex_mincount=count, ex_maxcount=count)
where count is usually between 500 and 2000. From my understanding of
the code it seems that the create_node method for ec2
(BaseEC2NodeDriver.create_method) uses an amazon RunInstances request
and is not based on NodeImage.
Thanks again,
Marko
On Wed, Jun 25, 2014 at 9:24 PM, Tomaz Muraus <to...@apache.org> wrote:
> Yeah, what others have said.
>
> By default, calling list_images in the EC2 driver without any filters
> returns a huge XML response. If lxml is not available (this feature is only
> available in 0.15.0), parsing this response takes a while and consumes a
> lot of memory.
>
> If you are not dynamically obtaining the image based on the name and you
> already know the id, I would recommend you to use the approach Jayy Vis has
> suggested.
>
> On Wed, Jun 25, 2014 at 7:13 AM, Jayy Vis <ja...@pickle.io> wrote:
>
>> create_node api of libcloud just uses image.id
>>
>> So you can simply create image object directly as follows.
>>
>> Driver = get_driver(Provider.EC2_US_EAST)
>> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>>
>> from libcloud.compute.base import NodeImage
>>
>> ec2_ami_id='ami-123'
>>
>> image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id, extra=None)
>>
>>
>>
>> On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <LE...@battelle.org>
>> wrote:
>>
>> > Do you guys have lxml installed? With the dev release it will use lxml
>> for
>> > parsing if installed and that should help a bit.
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
>> > Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
>> > To: users@libcloud.apache.org
>> > Subject: Re: list_images crashes (for ec2)
>> >
>> >
>> > Where are you actually running this code from? I have found in recent
>> > months that, when running libcloud against EC2 on a micro instance,
>> > list_images() returns just too many images for the machine to handle,
>> > and eventually the OOM killer destroys the process.
>> >
>> > On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <ma...@gmail.com>
>> > wrote:
>> > > Hello,
>> > >
>> > > The following script crashes when running on Amazon instance.
>> > >
>> > > ================================
>> > > from libcloud.compute.types import Provider
>> > > from libcloud.compute.providers import get_driver
>> > >
>> > > EC2_ACCESS_ID = ...
>> > > EC2_SECRET = ...
>> > > Driver = get_driver(Provider.EC2_US_EAST)
>> > > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>> > >
>> > > print "before"
>> > > conn.list_images()
>> > > print "after"
>> > > ================================
>> > >
>> > > The first print appears and "Killed" is displayed before the second
>> > print.
>> > >
>> > > I suspect it is an error in etree module (apparently the list of
>> > > available images is huge).
>> > >
>> > > I use list_images to get the python object of the desired image
>> > >
>> > > image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
>> > >
>> > > so that I can create a node with this image (create_node method)
>> > >
>> > > Do you know if there is a way to create a node without using
>> > > list_images or if there is another way to avoid this crash?
>> > >
>> > > I am using python 2.7.3 and libcloud 0.14.1.
>> > >
>> > > Thanks a lot in advance!
>> > > Marko
>> >
>> >
>> >
>> > --
>> > "In order to create, you have to have the willingness, the desire to
>> > be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
>> > 2011)
>> >
>>
Re: list_images crashes (for ec2)
Posted by Tomaz Muraus <to...@apache.org>.
Yeah, what others have said.
By default, calling list_images in the EC2 driver without any filters
returns a huge XML response. If lxml is not available (this feature is only
available in 0.15.0), parsing this response takes a while and consumes a
lot of memory.
If you are not dynamically obtaining the image based on the name and you
already know the id, I would recommend you to use the approach Jayy Vis has
suggested.
On Wed, Jun 25, 2014 at 7:13 AM, Jayy Vis <ja...@pickle.io> wrote:
> create_node api of libcloud just uses image.id
>
> So you can simply create image object directly as follows.
>
> Driver = get_driver(Provider.EC2_US_EAST)
> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>
> from libcloud.compute.base import NodeImage
>
> ec2_ami_id='ami-123'
>
> image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id, extra=None)
>
>
>
> On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <LE...@battelle.org>
> wrote:
>
> > Do you guys have lxml installed? With the dev release it will use lxml
> for
> > parsing if installed and that should help a bit.
> >
> >
> >
> > -----Original Message-----
> > From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
> > Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
> > To: users@libcloud.apache.org
> > Subject: Re: list_images crashes (for ec2)
> >
> >
> > Where are you actually running this code from? I have found in recent
> > months that, when running libcloud against EC2 on a micro instance,
> > list_images() returns just too many images for the machine to handle,
> > and eventually the OOM killer destroys the process.
> >
> > On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <ma...@gmail.com>
> > wrote:
> > > Hello,
> > >
> > > The following script crashes when running on Amazon instance.
> > >
> > > ================================
> > > from libcloud.compute.types import Provider
> > > from libcloud.compute.providers import get_driver
> > >
> > > EC2_ACCESS_ID = ...
> > > EC2_SECRET = ...
> > > Driver = get_driver(Provider.EC2_US_EAST)
> > > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> > >
> > > print "before"
> > > conn.list_images()
> > > print "after"
> > > ================================
> > >
> > > The first print appears and "Killed" is displayed before the second
> > print.
> > >
> > > I suspect it is an error in etree module (apparently the list of
> > > available images is huge).
> > >
> > > I use list_images to get the python object of the desired image
> > >
> > > image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
> > >
> > > so that I can create a node with this image (create_node method)
> > >
> > > Do you know if there is a way to create a node without using
> > > list_images or if there is another way to avoid this crash?
> > >
> > > I am using python 2.7.3 and libcloud 0.14.1.
> > >
> > > Thanks a lot in advance!
> > > Marko
> >
> >
> >
> > --
> > "In order to create, you have to have the willingness, the desire to
> > be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
> > 2011)
> >
>
Re: list_images crashes (for ec2)
Posted by Jayy Vis <ja...@pickle.io>.
create_node api of libcloud just uses image.id
So you can simply create image object directly as follows.
Driver = get_driver(Provider.EC2_US_EAST)
conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
from libcloud.compute.base import NodeImage
ec2_ami_id='ami-123'
image=NodeImage(driver=conn, id=ec2_ami_id, name=ec2_ami_id, extra=None)
On Wed, Jun 25, 2014 at 3:36 AM, Lehman, Matthew J <LE...@battelle.org>
wrote:
> Do you guys have lxml installed? With the dev release it will use lxml for
> parsing if installed and that should help a bit.
>
>
>
> -----Original Message-----
> From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
> Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
> To: users@libcloud.apache.org
> Subject: Re: list_images crashes (for ec2)
>
>
> Where are you actually running this code from? I have found in recent
> months that, when running libcloud against EC2 on a micro instance,
> list_images() returns just too many images for the machine to handle,
> and eventually the OOM killer destroys the process.
>
> On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <ma...@gmail.com>
> wrote:
> > Hello,
> >
> > The following script crashes when running on Amazon instance.
> >
> > ================================
> > from libcloud.compute.types import Provider
> > from libcloud.compute.providers import get_driver
> >
> > EC2_ACCESS_ID = ...
> > EC2_SECRET = ...
> > Driver = get_driver(Provider.EC2_US_EAST)
> > conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
> >
> > print "before"
> > conn.list_images()
> > print "after"
> > ================================
> >
> > The first print appears and "Killed" is displayed before the second
> print.
> >
> > I suspect it is an error in etree module (apparently the list of
> > available images is huge).
> >
> > I use list_images to get the python object of the desired image
> >
> > image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
> >
> > so that I can create a node with this image (create_node method)
> >
> > Do you know if there is a way to create a node without using
> > list_images or if there is another way to avoid this crash?
> >
> > I am using python 2.7.3 and libcloud 0.14.1.
> >
> > Thanks a lot in advance!
> > Marko
>
>
>
> --
> "In order to create, you have to have the willingness, the desire to
> be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
> 2011)
>
RE: list_images crashes (for ec2)
Posted by "Lehman, Matthew J" <LE...@battelle.org>.
Do you guys have lxml installed? With the dev release it will use lxml for parsing if installed and that should help a bit.
-----Original Message-----
From: Joseph Hall [perlhoser@gmail.com<ma...@gmail.com>]
Sent: Tuesday, June 24, 2014 05:52 PM Eastern Standard Time
To: users@libcloud.apache.org
Subject: Re: list_images crashes (for ec2)
Where are you actually running this code from? I have found in recent
months that, when running libcloud against EC2 on a micro instance,
list_images() returns just too many images for the machine to handle,
and eventually the OOM killer destroys the process.
On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <ma...@gmail.com> wrote:
> Hello,
>
> The following script crashes when running on Amazon instance.
>
> ================================
> from libcloud.compute.types import Provider
> from libcloud.compute.providers import get_driver
>
> EC2_ACCESS_ID = ...
> EC2_SECRET = ...
> Driver = get_driver(Provider.EC2_US_EAST)
> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>
> print "before"
> conn.list_images()
> print "after"
> ================================
>
> The first print appears and "Killed" is displayed before the second print.
>
> I suspect it is an error in etree module (apparently the list of
> available images is huge).
>
> I use list_images to get the python object of the desired image
>
> image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
>
> so that I can create a node with this image (create_node method)
>
> Do you know if there is a way to create a node without using
> list_images or if there is another way to avoid this crash?
>
> I am using python 2.7.3 and libcloud 0.14.1.
>
> Thanks a lot in advance!
> Marko
--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)
Re: list_images crashes (for ec2)
Posted by Joseph Hall <pe...@gmail.com>.
Where are you actually running this code from? I have found in recent
months that, when running libcloud against EC2 on a micro instance,
list_images() returns just too many images for the machine to handle,
and eventually the OOM killer destroys the process.
On Tue, Jun 24, 2014 at 3:33 PM, Marko Loparic <ma...@gmail.com> wrote:
> Hello,
>
> The following script crashes when running on Amazon instance.
>
> ================================
> from libcloud.compute.types import Provider
> from libcloud.compute.providers import get_driver
>
> EC2_ACCESS_ID = ...
> EC2_SECRET = ...
> Driver = get_driver(Provider.EC2_US_EAST)
> conn = Driver(EC2_ACCESS_ID, EC2_SECRET)
>
> print "before"
> conn.list_images()
> print "after"
> ================================
>
> The first print appears and "Killed" is displayed before the second print.
>
> I suspect it is an error in etree module (apparently the list of
> available images is huge).
>
> I use list_images to get the python object of the desired image
>
> image = [i for i in conn.list_images() if i.id == EC2_AMI][0]
>
> so that I can create a node with this image (create_node method)
>
> Do you know if there is a way to create a node without using
> list_images or if there is another way to avoid this crash?
>
> I am using python 2.7.3 and libcloud 0.14.1.
>
> Thanks a lot in advance!
> Marko
--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)