You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@vcl.apache.org by Michael Jinks <mj...@uchicago.edu> on 2012/10/03 19:30:51 UTC
Block allocations always fail
Hi List.
We have a VCL 2.2 instance running, most things work, but one glaring
problem is still thwarting me. When I try to create a block allocation,
the computers always come up "Failed"; this in spite of the fact that I
can reserve the same image individually with no issues.
I've tried this so far with the admin@Local account, as well as my user
account which has full admin privileges (or at least as close as we can
get by assigning rights in the Privileges page).
I go to the "Block Allocations" page, click "Create New Block
Allocation", fill in the form, choosing a known-working image in the
"Environment" dropdown, and a "List of Dates/Times" with one time span
in the near future. "Submit New Block Allocation" pops up a
confirmation window, that goes fine, and I have an entry "Block
Allocations" page under "Manage Block Allocations". Looks fine.
But, if I click an entry under "Your Active Block Allocations", under
"Current status of computers," the number by "Failed" is equal to the
number of seats I reserved, and as far as I can tell the system never
tries to deploy a VM.
I'm guessing that there is some permission setting or some other item
within the management node that I've overlooked. Any hints on where to
look?
Thanks,
-j
--
Michael Jinks :: mjinks@uchicago.edu
University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
On Mon, Oct 15, 2012 at 02:07:24PM -0500, Michael Jinks wrote:
> On Mon, Oct 15, 2012 at 02:02:25PM -0500, Michael Jinks wrote:
> > Yes: my database is messed up. Apparently when I re-added the
> > 'vclsystem' account, 'vcld -setup' didn't stuff a line into the
> > localauth table. I'm about to do that by hand and see what happens.
>
> Okay! Now, vclsystem can RPC to the web interface and get a printout of
> the supported methods.
>
> Now back to trying block reservations.
Worky worky!
Thanks, one and all. I've just watched my first successful block
reservation go through.
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
On Mon, Oct 15, 2012 at 02:02:25PM -0500, Michael Jinks wrote:
> Yes: my database is messed up. Apparently when I re-added the
> 'vclsystem' account, 'vcld -setup' didn't stuff a line into the
> localauth table. I'm about to do that by hand and see what happens.
Okay! Now, vclsystem can RPC to the web interface and get a printout of
the supported methods.
Now back to trying block reservations.
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Yes: my database is messed up. Apparently when I re-added the
'vclsystem' account, 'vcld -setup' didn't stuff a line into the
localauth table. I'm about to do that by hand and see what happens.
I'm looking at the SQL setup script from the VCL code distribution but
it's easy to overlook things so I'll ask: Are there other steps I'll
need to take in order to recreate a functional vclsystem account?
On Mon, Oct 15, 2012 at 11:21:41AM -0500, Michael Jinks wrote:
> On Mon, Oct 15, 2012 at 12:14:58PM -0400, Andy Kurth wrote:
> > Can you login to the VCL website as vclsystem?
>
> Nope. But this was the first it occurred to me that the web UI should
> work for that account. I'd been thinking of it as unique to the xmlrpc
> stuff, for some reason.
>
> > Also try running the
> > test script as admin instead of vclsystem. Same result?
>
> That works! Hrm. Okay, so yeah, my vclsystem account is plainly hosed,
> somehow. I'll go back to the doc Aaron linked for updating passwords
> and see if that helps.
>
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
On Mon, Oct 15, 2012 at 12:14:58PM -0400, Andy Kurth wrote:
> Can you login to the VCL website as vclsystem?
Nope. But this was the first it occurred to me that the web UI should
work for that account. I'd been thinking of it as unique to the xmlrpc
stuff, for some reason.
> Also try running the
> test script as admin instead of vclsystem. Same result?
That works! Hrm. Okay, so yeah, my vclsystem account is plainly hosed,
somehow. I'll go back to the doc Aaron linked for updating passwords
and see if that helps.
Re: Block allocations and Shib, was Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Oh criminy. Yep, that's what I did.
Thanks.
Sigh.
On Tue, Oct 16, 2012 at 02:52:27PM -0400, Josh Thompson wrote:
> Did you require Shib authentication on the vcl directory? You should not
> require it there, but on the vcl/shibauth directory instead.
>
> Josh
>
> On Tuesday, October 16, 2012 12:33:07 PM Michael Jinks wrote:
> > Well I don't know why it took me so long to notice this.
> >
> > We use Shibboleth to provide authN for our VCL web interfaces, and
> > during debugging we sometimes switch it off to make it easier to get at
> > the local admin account. I've just realized that this has been
> > contributing to our block allocation problems. When Shib is disabled,
> > bock allocations are working. But when it's switched back on, the
> > vclsystem account is intercepted by Apache's Shib module and never has
> > a chance to talk to the real web interface.
> >
> > I figure this must have come up elsewhere; how are other sites working
> > around this issue?
> >
> > We have a project on our timeline to put in a back door for admin
> > access to VCL, bypassing Shib, but we haven't started work on it yet
> > and I don't know if the XML/RPC interface will pose its own issues
> > separate from what we'll need to deal with for browser access.
> >
> > Thanks as always,
> > --Michael
> --
> -------------------------------
> Josh Thompson
> Systems Programmer
> Advanced Computing | VCL Developer
> North Carolina State University
>
> Josh_Thompson@ncsu.edu
> 919-515-5323
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
--
Michael Jinks :: mjinks@uchicago.edu
University of Chicago IT Services
Re: Block allocations and Shib, was Re: Block allocations always fail
Posted by Josh Thompson <jo...@ncsu.edu>.
Did you require Shib authentication on the vcl directory? You should not
require it there, but on the vcl/shibauth directory instead.
Josh
On Tuesday, October 16, 2012 12:33:07 PM Michael Jinks wrote:
> Well I don't know why it took me so long to notice this.
>
> We use Shibboleth to provide authN for our VCL web interfaces, and
> during debugging we sometimes switch it off to make it easier to get at
> the local admin account. I've just realized that this has been
> contributing to our block allocation problems. When Shib is disabled,
> bock allocations are working. But when it's switched back on, the
> vclsystem account is intercepted by Apache's Shib module and never has
> a chance to talk to the real web interface.
>
> I figure this must have come up elsewhere; how are other sites working
> around this issue?
>
> We have a project on our timeline to put in a back door for admin
> access to VCL, bypassing Shib, but we haven't started work on it yet
> and I don't know if the XML/RPC interface will pose its own issues
> separate from what we'll need to deal with for browser access.
>
> Thanks as always,
> --Michael
--
-------------------------------
Josh Thompson
Systems Programmer
Advanced Computing | VCL Developer
North Carolina State University
Josh_Thompson@ncsu.edu
919-515-5323
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: Block allocations and Shib, was Re: Block allocations always
fail
Posted by Aaron Coburn <ac...@amherst.edu>.
Shibboleth, LDAP and local authentication are parallel authentication mechanisms. They do not interfere with each other.
> We use Shibboleth to provide authN for our VCL web interfaces, and
> during debugging we sometimes switch it off to make it easier to get at
> the local admin account. I've just realized that this has been
> contributing to our block allocation problems. When Shib is disabled,
> bock allocations are working. But when it's switched back on, the
> vclsystem account is intercepted by Apache's Shib module and never has
> a chance to talk to the real web interface.
There are two reasons why this issue could emerge: either your entire vcl web application is being protected by shibboleth or the web browser is sending a shibboleth cookie in its GET request.
If it is the first issue, change the apache httpd configuration so that ONLY the shibauth/ directory is protected by shibboleth. Then restart httpd.
If it is the second issue, you need to restart, i.e. completely quit, your web browser (or use a secondary web browser -- not just a new window). Shibboleth works in a browser by sending session cookies: the VCL web interface inspects these cookies in the HTTP request headers, and if there is a shib cookie present, the browser is redirected to the shibauth/ directory for any user without an otherwise valid application-level cookie. This happens, for instance, when you login to the VCL by means of a Shibboleth IdP, then try to logout, and then go back to the VCL site to login.
I will note that logout is complicated when Shibboleth is involved; if you want Shibboleth logout to work with the VCL, you will need to perform a number of customizations, ensuring that every shib cookie is destroyed or invalidated during the logout process. More information about this is available here: http://people.apache.org/~acoburn/shibboleth.html
> I figure this must have come up elsewhere; how are other sites working
> around this issue?
There is no need for a work-around. These different mechanisms can exist just fine in parallel.
> We have a project on our timeline to put in a back door for admin
> access to VCL, bypassing Shib, but we haven't started work on it yet
> and I don't know if the XML/RPC interface will pose its own issues
> separate from what we'll need to deal with for browser access.
You shouldn't need a special 'back door' for admin access -- just use the 'admin@local' or some other @local account. Those @local accounts are exceedingly handy for both administrative and automated access to the VCL.
It should also be clear that the XML/RPC interface does not work for shib-authenticated users: the VCL application does not, itself, have any means for validating a shib-based user's password, so there is no way for the user to authenticate. More information about this is available here: https://issues.apache.org/jira/browse/VCL-608 --- the way around this is something called delegated authentication.
Aaron
--
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
acoburn@amherst.edu
Block allocations and Shib, was Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Well I don't know why it took me so long to notice this.
We use Shibboleth to provide authN for our VCL web interfaces, and
during debugging we sometimes switch it off to make it easier to get at
the local admin account. I've just realized that this has been
contributing to our block allocation problems. When Shib is disabled,
bock allocations are working. But when it's switched back on, the
vclsystem account is intercepted by Apache's Shib module and never has
a chance to talk to the real web interface.
I figure this must have come up elsewhere; how are other sites working
around this issue?
We have a project on our timeline to put in a back door for admin
access to VCL, bypassing Shib, but we haven't started work on it yet
and I don't know if the XML/RPC interface will pose its own issues
separate from what we'll need to deal with for browser access.
Thanks as always,
--Michael
--
Michael Jinks :: mjinks@uchicago.edu
University of Chicago IT Services
Re: Block allocations always fail
Posted by Andy Kurth <an...@ncsu.edu>.
Can you login to the VCL website as vclsystem? Also try running the test
script as admin instead of vclsystem. Same result?
On Mon, Oct 15, 2012 at 12:00 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> Hi Andy.
>
> I get:
>
> $ ~/test-xmlrpc.pl
> Username: vclsystem
> Password: created RPC::XML client object RPC::XML::Client::send_request
> fault occurred, fault code: 3, fault string: Access denied
>
> ...which looks like I'm entering the wrong password, but I'm pretty sure
> that's not the case.
>
> vcld.log doesn't show anything in response to the attempt but I don't
> know if it should.
>
> I wonder if my database is hosed (again, still).
>
>
>
> On Mon, Oct 15, 2012 at 11:31:36AM -0400, Andy Kurth wrote:
> > What output does the following script produce? You should only have
> to
> > change the URL. (Made minor changes to Aaron's)
> >
> > #---BEGIN---
> >
> > #!/usr/bin/perl -w
> > use strict;
> > use warnings;
> > use diagnostics;
> > use RPC::XML::Client;
> > use Term::ReadKey;
> > use Data::Dumper;
> > my $VCL_LOCATION = 'https://<URL>/index.php?mode=xmlrpccall';
> > $|++;
> > print "Username: ";
> > chomp(my $username = <>);
> > print "Password: ";
> > ReadMode 2;
> > chomp(my $password = <>);
> > ReadMode 0;
> > my $client = RPC::XML::Client->new($VCL_LOCATION, useragent =>
> > ['ssl_opts' => {verify_hostname => 0}]);
> > if (defined($client)) {
> > print "created RPC::XML client object\n";
> > }
> > else {
> > print "failed to create a new RPC::XML client object, error: " .
> > ($RPC::XML::ERROR || '<none>') . "\n";
> > exit;
> > }
> > $client->useragent->default_header('X-User' => $username);
> > $client->useragent->default_header('X-Pass' => $password);
> > $client->useragent->default_header('X-APIVERSION' => 2);
> > my $response = $client->send_request(("system.listMethods"));
> > if (!ref($response)) {
> > print "RPC::XML::Client::send_request failed, error: " .
> > ($RPC::XML::ERROR || '<none>') . "\n" . Dumper($response) . "\n";
> > exit;
> > }
> > elsif ($response->is_fault) {
> > print "RPC::XML::Client::send_request fault occurred, fault code: " .
> > $response->code . ", fault string: " . $response->string . "\n";
> > exit;
> > }
> > for my $method (@$response) {
> > print $method->value . "\n";
> > }
> >
> > #---END---
> >
> > On Fri, Oct 12, 2012 at 5:54 PM, Michael Jinks <[1]
> mjinks@uchicago.edu>
> > wrote:
> >
> > On Fri, Oct 12, 2012 at 09:36:08PM +0000, Aaron Coburn wrote:
> > >
> > > > Oh, good! I was wishing for something like this.
> > > >
> > > > This looks familiar:
> > > >
> > > > $ ./[2]vcl-rpcxml.pl
> > > > Username: vclsystem
> > > > Password:
> > > > Can't locate object method "value" via package
> > > > "RPC::XML::Client::send_request: HTTP server error: Can't connect
> > to
> > > > [3]vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps
> > you forgot
> > > > to load "RPC::XML::Client::send_request: HTTP server error: Can't
> > > > connect to [4]vlab-a.uchicago.edu:443 (certificate verify
> > failed)"?) at
> > > > ./[5]vcl-rpcxml.pl line 28, <> line 2.
> > > >
> > > > I get the same result whether I enter the correct password or
> > something
> > > > deliberately wrong. So I'm guessing that this is indicating
> > something
> > > > missing in my environment, but I know from looking previously that
> > the
> > > > RPC::XML::Client directory is complete with a send_request script.
> > > >
> > >
> > > But are you not using a custom-built perl interpreter? You should
> > invoke the script with the same perl interpreter that vcld is using;
> > otherwise, it will use /usr/bin/perl
> >
> > I adjusted the script:
> > #!/usr/local/vcl/perl/bin/perl -w
> >
> > > The error you see relates to a package not being installed -- either
> > LWP or RPC::XML
> >
> > Yeah, that sure is what it looks like... but both modules are
> > present:
> > $ sudo /usr/local/vcl/perl/bin/cpan LWP
> > Reading '/root/.cpan/Metadata'
> > Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> > LWP is up to date (6.04).
> > $ sudo /usr/local/vcl/perl/bin/cpan RPC::XML
> > Reading '/root/.cpan/Metadata'
> > Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> > RPC::XML is up to date (1.56).
> >
> > --
> > Michael Jinks :: [6]mjinks@uchicago.edu
> > University of Chicago IT Services
> >
> > References
> >
> > 1. mailto:mjinks@uchicago.edu
> > 2. http://vcl-rpcxml.pl/
> > 3. http://vlab-a.uchicago.edu:443/
> > 4. http://vlab-a.uchicago.edu:443/
> > 5. http://vcl-rpcxml.pl/
> > 6. mailto:mjinks@uchicago.edu
>
> --
> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
>
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Hi Andy.
I get:
$ ~/test-xmlrpc.pl
Username: vclsystem
Password: created RPC::XML client object RPC::XML::Client::send_request fault occurred, fault code: 3, fault string: Access denied
...which looks like I'm entering the wrong password, but I'm pretty sure
that's not the case.
vcld.log doesn't show anything in response to the attempt but I don't
know if it should.
I wonder if my database is hosed (again, still).
On Mon, Oct 15, 2012 at 11:31:36AM -0400, Andy Kurth wrote:
> What output does the following script produce? You should only have to
> change the URL. (Made minor changes to Aaron's)
>
> #---BEGIN---
>
> #!/usr/bin/perl -w
> use strict;
> use warnings;
> use diagnostics;
> use RPC::XML::Client;
> use Term::ReadKey;
> use Data::Dumper;
> my $VCL_LOCATION = 'https://<URL>/index.php?mode=xmlrpccall';
> $|++;
> print "Username: ";
> chomp(my $username = <>);
> print "Password: ";
> ReadMode 2;
> chomp(my $password = <>);
> ReadMode 0;
> my $client = RPC::XML::Client->new($VCL_LOCATION, useragent =>
> ['ssl_opts' => {verify_hostname => 0}]);
> if (defined($client)) {
> print "created RPC::XML client object\n";
> }
> else {
> print "failed to create a new RPC::XML client object, error: " .
> ($RPC::XML::ERROR || '<none>') . "\n";
> exit;
> }
> $client->useragent->default_header('X-User' => $username);
> $client->useragent->default_header('X-Pass' => $password);
> $client->useragent->default_header('X-APIVERSION' => 2);
> my $response = $client->send_request(("system.listMethods"));
> if (!ref($response)) {
> print "RPC::XML::Client::send_request failed, error: " .
> ($RPC::XML::ERROR || '<none>') . "\n" . Dumper($response) . "\n";
> exit;
> }
> elsif ($response->is_fault) {
> print "RPC::XML::Client::send_request fault occurred, fault code: " .
> $response->code . ", fault string: " . $response->string . "\n";
> exit;
> }
> for my $method (@$response) {
> print $method->value . "\n";
> }
>
> #---END---
>
> On Fri, Oct 12, 2012 at 5:54 PM, Michael Jinks <[1...@uchicago.edu>
> wrote:
>
> On Fri, Oct 12, 2012 at 09:36:08PM +0000, Aaron Coburn wrote:
> >
> > > Oh, good! I was wishing for something like this.
> > >
> > > This looks familiar:
> > >
> > > $ ./[2]vcl-rpcxml.pl
> > > Username: vclsystem
> > > Password:
> > > Can't locate object method "value" via package
> > > "RPC::XML::Client::send_request: HTTP server error: Can't connect
> to
> > > [3]vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps
> you forgot
> > > to load "RPC::XML::Client::send_request: HTTP server error: Can't
> > > connect to [4]vlab-a.uchicago.edu:443 (certificate verify
> failed)"?) at
> > > ./[5]vcl-rpcxml.pl line 28, <> line 2.
> > >
> > > I get the same result whether I enter the correct password or
> something
> > > deliberately wrong. So I'm guessing that this is indicating
> something
> > > missing in my environment, but I know from looking previously that
> the
> > > RPC::XML::Client directory is complete with a send_request script.
> > >
> >
> > But are you not using a custom-built perl interpreter? You should
> invoke the script with the same perl interpreter that vcld is using;
> otherwise, it will use /usr/bin/perl
>
> I adjusted the script:
> #!/usr/local/vcl/perl/bin/perl -w
>
> > The error you see relates to a package not being installed -- either
> LWP or RPC::XML
>
> Yeah, that sure is what it looks like... but both modules are
> present:
> $ sudo /usr/local/vcl/perl/bin/cpan LWP
> Reading '/root/.cpan/Metadata'
> Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> LWP is up to date (6.04).
> $ sudo /usr/local/vcl/perl/bin/cpan RPC::XML
> Reading '/root/.cpan/Metadata'
> Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> RPC::XML is up to date (1.56).
>
> --
> Michael Jinks :: [6]mjinks@uchicago.edu
> University of Chicago IT Services
>
> References
>
> 1. mailto:mjinks@uchicago.edu
> 2. http://vcl-rpcxml.pl/
> 3. http://vlab-a.uchicago.edu:443/
> 4. http://vlab-a.uchicago.edu:443/
> 5. http://vcl-rpcxml.pl/
> 6. mailto:mjinks@uchicago.edu
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Andy Kurth <an...@ncsu.edu>.
What output does the following script produce? You should only have to
change the URL. (Made minor changes to Aaron's)
#---BEGIN---
#!/usr/bin/perl -w
use strict;
use warnings;
use diagnostics;
use RPC::XML::Client;
use Term::ReadKey;
use Data::Dumper;
my $VCL_LOCATION = 'https://<URL>/index.php?mode=xmlrpccall';
$|++;
print "Username: ";
chomp(my $username = <>);
print "Password: ";
ReadMode 2;
chomp(my $password = <>);
ReadMode 0;
my $client = RPC::XML::Client->new($VCL_LOCATION, useragent => ['ssl_opts'
=> {verify_hostname => 0}]);
if (defined($client)) {
print "created RPC::XML client object\n";
}
else {
print "failed to create a new RPC::XML client object, error: " .
($RPC::XML::ERROR || '<none>') . "\n";
exit;
}
$client->useragent->default_header('X-User' => $username);
$client->useragent->default_header('X-Pass' => $password);
$client->useragent->default_header('X-APIVERSION' => 2);
my $response = $client->send_request(("system.listMethods"));
if (!ref($response)) {
print "RPC::XML::Client::send_request failed, error: " . ($RPC::XML::ERROR
|| '<none>') . "\n" . Dumper($response) . "\n";
exit;
}
elsif ($response->is_fault) {
print "RPC::XML::Client::send_request fault occurred, fault code: " .
$response->code . ", fault string: " . $response->string . "\n";
exit;
}
for my $method (@$response) {
print $method->value . "\n";
}
#---END---
On Fri, Oct 12, 2012 at 5:54 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> On Fri, Oct 12, 2012 at 09:36:08PM +0000, Aaron Coburn wrote:
> >
> > > Oh, good! I was wishing for something like this.
> > >
> > > This looks familiar:
> > >
> > > $ ./vcl-rpcxml.pl
> > > Username: vclsystem
> > > Password:
> > > Can't locate object method "value" via package
> > > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> > > vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you
> forgot
> > > to load "RPC::XML::Client::send_request: HTTP server error: Can't
> > > connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at
> > > ./vcl-rpcxml.pl line 28, <> line 2.
> > >
> > > I get the same result whether I enter the correct password or something
> > > deliberately wrong. So I'm guessing that this is indicating something
> > > missing in my environment, but I know from looking previously that the
> > > RPC::XML::Client directory is complete with a send_request script.
> > >
> >
> > But are you not using a custom-built perl interpreter? You should invoke
> the script with the same perl interpreter that vcld is using; otherwise, it
> will use /usr/bin/perl
>
> I adjusted the script:
>
> #!/usr/local/vcl/perl/bin/perl -w
>
> > The error you see relates to a package not being installed -- either LWP
> or RPC::XML
>
> Yeah, that sure is what it looks like... but both modules are present:
>
> $ sudo /usr/local/vcl/perl/bin/cpan LWP
> Reading '/root/.cpan/Metadata'
> Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> LWP is up to date (6.04).
>
> $ sudo /usr/local/vcl/perl/bin/cpan RPC::XML
> Reading '/root/.cpan/Metadata'
> Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
> RPC::XML is up to date (1.56).
>
>
> --
> Michael Jinks :: mjinks@uchicago.edu
> University of Chicago IT Services
>
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
On Fri, Oct 12, 2012 at 09:36:08PM +0000, Aaron Coburn wrote:
>
> > Oh, good! I was wishing for something like this.
> >
> > This looks familiar:
> >
> > $ ./vcl-rpcxml.pl
> > Username: vclsystem
> > Password:
> > Can't locate object method "value" via package
> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> > vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot
> > to load "RPC::XML::Client::send_request: HTTP server error: Can't
> > connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at
> > ./vcl-rpcxml.pl line 28, <> line 2.
> >
> > I get the same result whether I enter the correct password or something
> > deliberately wrong. So I'm guessing that this is indicating something
> > missing in my environment, but I know from looking previously that the
> > RPC::XML::Client directory is complete with a send_request script.
> >
>
> But are you not using a custom-built perl interpreter? You should invoke the script with the same perl interpreter that vcld is using; otherwise, it will use /usr/bin/perl
I adjusted the script:
#!/usr/local/vcl/perl/bin/perl -w
> The error you see relates to a package not being installed -- either LWP or RPC::XML
Yeah, that sure is what it looks like... but both modules are present:
$ sudo /usr/local/vcl/perl/bin/cpan LWP
Reading '/root/.cpan/Metadata'
Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
LWP is up to date (6.04).
$ sudo /usr/local/vcl/perl/bin/cpan RPC::XML
Reading '/root/.cpan/Metadata'
Database was generated on Fri, 12 Oct 2012 09:07:03 GMT
RPC::XML is up to date (1.56).
--
Michael Jinks :: mjinks@uchicago.edu
University of Chicago IT Services
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
> Oh, good! I was wishing for something like this.
>
> This looks familiar:
>
> $ ./vcl-rpcxml.pl
> Username: vclsystem
> Password:
> Can't locate object method "value" via package
> "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot
> to load "RPC::XML::Client::send_request: HTTP server error: Can't
> connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at
> ./vcl-rpcxml.pl line 28, <> line 2.
>
> I get the same result whether I enter the correct password or something
> deliberately wrong. So I'm guessing that this is indicating something
> missing in my environment, but I know from looking previously that the
> RPC::XML::Client directory is complete with a send_request script.
>
But are you not using a custom-built perl interpreter? You should invoke the script with the same perl interpreter that vcld is using; otherwise, it will use /usr/bin/perl
The error you see relates to a package not being installed -- either LWP or RPC::XML
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
On Fri, Oct 12, 2012 at 07:47:59PM +0000, Aaron Coburn wrote:
> In .ht-inc/conf.php, have you, by any chance, changed DEFAULT_AFFILID? (It should be '1' and it should correspond to the local affiliation id).
Nope, no changes there. It's still set to '1'.
> Also, have you possibly removed the "Local Account" entry in the $authMechs array? (You should have an entry with type => local and affiliationid => 1).
That looks good too. All we've changed is the "Help" entry:
"Local Account" => array("type" => "local",
"affiliationid" => 1,
"help" => "You probably don't want \"Local Account\" unless you are are an administrator of the VCL system."),
> Some time ago, I wrote a perl script for testing access to the remote API. I have put it online here: http://people.apache.org/~acoburn/rpcxml.pl
Oh, good! I was wishing for something like this.
This looks familiar:
$ ./vcl-rpcxml.pl
Username: vclsystem
Password:
Can't locate object method "value" via package
"RPC::XML::Client::send_request: HTTP server error: Can't connect to
vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot
to load "RPC::XML::Client::send_request: HTTP server error: Can't
connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at
./vcl-rpcxml.pl line 28, <> line 2.
I get the same result whether I enter the correct password or something
deliberately wrong. So I'm guessing that this is indicating something
missing in my environment, but I know from looking previously that the
RPC::XML::Client directory is complete with a send_request script.
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
In .ht-inc/conf.php, have you, by any chance, changed DEFAULT_AFFILID? (It should be '1' and it should correspond to the local affiliation id).
Also, have you possibly removed the "Local Account" entry in the $authMechs array? (You should have an entry with type => local and affiliationid => 1).
Some time ago, I wrote a perl script for testing access to the remote API. I have put it online here: http://people.apache.org/~acoburn/rpcxml.pl
You will need to modify the script with the correct location of your VCL system, but otherwise, this will allow you to test access to the API from your management node with the vclsystem user. Try using the 'vclsystem' user account with the password specified in /etc/vcl/vcld.conf
-Aaron
On Oct 12, 2012, at 2:57 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> Here's my latest. I believe I've sorted out my trouble with the xmlrpc
> user account (cleared out foreign key references it all my tables, then
> changed its user ID to 3), and I'm running with Aaron Coburn's suggested
> changs to utils.pm.
>
> I'm pasting the entire log from my latest block reseration attempt
> below. I'm getting more information but I still don't see anything that
> looks like a smoking gun to me.
>
>
>
> 2012-10-12 13:42:16|2375|utils.pm:check_blockrequest_time(1026)|block request start time is within start window (32 minutes from now),
> returning 'start'
> 2012-10-12 13:42:16|2375|DataStructure.pm:_automethod(834)|data structure updated: $self->blockrequest_data->{8}{MODE}
> |2375| blockrequest_mode = start
> 2012-10-12 13:42:16|2375|vcld:main(445)|block request 8 'michael blocks three machines' processing set to 1
> 2012-10-12 13:42:16|2375|vcld:make_new_child(515)|loaded VCL::blockrequest module
> 2012-10-12 13:42:16|2375|vcld:make_new_child(539)|current number of forked kids: 1
> 2012-10-12 13:42:16|4763|vcld:make_new_child(555)|vcld environment variable set to 0 for this process
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'numMachines' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'ownerid' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'expireTime' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'repeating' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'admingroupid' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'status' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'managementnodeid' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'name' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'blockTimes' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'groupname' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'id' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'groupid' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'imageid' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'processing' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'MODE' key for VCL::blockrequest object from arguments
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:create_mn_os_object(361)|VCL::Module::OS::Linux::ManagementNode module loaded
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(196)|VCL::Module::OS::Linux::ManagementNode object created for image <not set>, ad
> dress: 23934e8
> 2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
> computer}{hostname}
> |4763|blockrequest| computer_hostname = vlab-a.uchicago.edu
> 2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
> computer}{NODENAME}
> |4763|blockrequest| computer_node_name = vlab-a
> 2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
> computer}{SHORTNAME}
> |4763|blockrequest| computer_short_name = vlab-a
> 2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
> computer}{IPaddress}
> |4763|blockrequest| computer_ip_address = 10.50.114.20
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:create_mn_os_object(366)|VCL::Module::OS::Linux::ManagementNode OS object created, address: 23934e8
> 2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(192)|VCL::blockrequest object created for state <not set>, address: 213c6c0
> 2012-10-12 13:42:16|4763|blockrequest|blockrequest.pm:initialize(84)|obtained a database handle for this state process, stored as $ENV{dbh}
> 2012-10-12 13:42:16|4763|blockrequest|utils.pm:rename_vcld_process(7136)|renamed process to 'vcld 8:11 'michael blocks three machines''
> 2012-10-12 13:42:16|4763|blockrequest|blockrequest.pm:initialize(94)|returning 1
> 2012-10-12 13:42:16|4763|blockrequest|vcld:make_new_child(565)|VCL::blockrequest object created and initialized
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(158)|blockrequest id: 8
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(159)|blockrequest mode: start
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(160)|blockrequest image id: 90
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(161)|blockrequest number machines: 3
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(162)|blockrequest expire: 2012-10-12 23:59:59
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(163)|blocktime id: 11
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(164)|blocktime processed: 0
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(165)|blocktime start: 2012-10-12 14:15:00
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(166)|owner email: vcl-system@lists.uchicago.edu
> Use of uninitialized value $owner_affiliation_helpaddress in concatenation (.)
> or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 167 (#2)
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $owner_affiliation_helpaddress in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 167.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 167)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(167)|help address:
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 11
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 1
> 2012-10-12 13:42:17|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
> |4763|blockrequest| Response class = RPC::XML::fault
> |4763|blockrequest| Response type = fault
> |4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
> |4763|blockrequest| Response value = HASH(0x294e428)
> |4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
> |4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
> |4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-3) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-4) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
> |4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
> 2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 2
> 2012-10-12 13:42:22|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
> |4763|blockrequest| Response class = RPC::XML::fault
> |4763|blockrequest| Response type = fault
> |4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
> |4763|blockrequest| Response value = HASH(0x2a79cf8)
> |4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
> |4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
> |4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-3) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-4) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
> |4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
> 2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 3
> 2012-10-12 13:42:27|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
> |4763|blockrequest| Response class = RPC::XML::fault
> |4763|blockrequest| Response type = fault
> |4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
> |4763|blockrequest| Response value = HASH(0x294e7a0)
> |4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
> |4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
> |4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-3) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-4) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
> |4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
> 2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 4
> 2012-10-12 13:42:32|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
> |4763|blockrequest| Response class = RPC::XML::fault
> |4763|blockrequest| Response type = fault
> |4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
> |4763|blockrequest| Response value = HASH(0x294e6b0)
> |4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
> |4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
> |4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-3) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-4) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
> |4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
> 2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 5
> 2012-10-12 13:42:37|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
> |4763|blockrequest| Response class = RPC::XML::fault
> |4763|blockrequest| Response type = fault
> |4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
> |4763|blockrequest| Response value = HASH(0x28976f0)
> |4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
> |4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
> |4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-3) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-4) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
> |4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
>
> |4763|blockrequest| ---- WARNING ----
> |4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
> |4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
> |4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
> |4763|blockrequest| (-2) vcld, make_new_child (line: 568)
> |4763|blockrequest| (-3) vcld, main (line: 448)
>
> 2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
>
>
>
>
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Here's my latest. I believe I've sorted out my trouble with the xmlrpc
user account (cleared out foreign key references it all my tables, then
changed its user ID to 3), and I'm running with Aaron Coburn's suggested
changs to utils.pm.
I'm pasting the entire log from my latest block reseration attempt
below. I'm getting more information but I still don't see anything that
looks like a smoking gun to me.
2012-10-12 13:42:16|2375|utils.pm:check_blockrequest_time(1026)|block request start time is within start window (32 minutes from now),
returning 'start'
2012-10-12 13:42:16|2375|DataStructure.pm:_automethod(834)|data structure updated: $self->blockrequest_data->{8}{MODE}
|2375| blockrequest_mode = start
2012-10-12 13:42:16|2375|vcld:main(445)|block request 8 'michael blocks three machines' processing set to 1
2012-10-12 13:42:16|2375|vcld:make_new_child(515)|loaded VCL::blockrequest module
2012-10-12 13:42:16|2375|vcld:make_new_child(539)|current number of forked kids: 1
2012-10-12 13:42:16|4763|vcld:make_new_child(555)|vcld environment variable set to 0 for this process
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'numMachines' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'ownerid' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'expireTime' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'repeating' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'admingroupid' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'status' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'managementnodeid' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'name' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'blockTimes' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'groupname' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'id' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'groupid' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'imageid' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'processing' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(172)|set 'MODE' key for VCL::blockrequest object from arguments
2012-10-12 13:42:16|4763|blockrequest|Module.pm:create_mn_os_object(361)|VCL::Module::OS::Linux::ManagementNode module loaded
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(196)|VCL::Module::OS::Linux::ManagementNode object created for image <not set>, ad
dress: 23934e8
2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
computer}{hostname}
|4763|blockrequest| computer_hostname = vlab-a.uchicago.edu
2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
computer}{NODENAME}
|4763|blockrequest| computer_node_name = vlab-a
2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
computer}{SHORTNAME}
|4763|blockrequest| computer_short_name = vlab-a
2012-10-12 13:42:16|4763|blockrequest|DataStructure.pm:_automethod(834)|data structure updated: $self->request_data->{reservation}{0}{
computer}{IPaddress}
|4763|blockrequest| computer_ip_address = 10.50.114.20
2012-10-12 13:42:16|4763|blockrequest|Module.pm:create_mn_os_object(366)|VCL::Module::OS::Linux::ManagementNode OS object created, address: 23934e8
2012-10-12 13:42:16|4763|blockrequest|Module.pm:new(192)|VCL::blockrequest object created for state <not set>, address: 213c6c0
2012-10-12 13:42:16|4763|blockrequest|blockrequest.pm:initialize(84)|obtained a database handle for this state process, stored as $ENV{dbh}
2012-10-12 13:42:16|4763|blockrequest|utils.pm:rename_vcld_process(7136)|renamed process to 'vcld 8:11 'michael blocks three machines''
2012-10-12 13:42:16|4763|blockrequest|blockrequest.pm:initialize(94)|returning 1
2012-10-12 13:42:16|4763|blockrequest|vcld:make_new_child(565)|VCL::blockrequest object created and initialized
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(158)|blockrequest id: 8
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(159)|blockrequest mode: start
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(160)|blockrequest image id: 90
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(161)|blockrequest number machines: 3
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(162)|blockrequest expire: 2012-10-12 23:59:59
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(163)|blocktime id: 11
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(164)|blocktime processed: 0
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(165)|blocktime start: 2012-10-12 14:15:00
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(166)|owner email: vcl-system@lists.uchicago.edu
Use of uninitialized value $owner_affiliation_helpaddress in concatenation (.)
or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 167 (#2)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $owner_affiliation_helpaddress in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 167.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 167)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(167)|help address:
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 11
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 1
2012-10-12 13:42:17|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
|4763|blockrequest| Response class = RPC::XML::fault
|4763|blockrequest| Response type = fault
|4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
|4763|blockrequest| Response value = HASH(0x294e428)
|4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
|4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
|4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-3) vcld, make_new_child (line: 568)
|4763|blockrequest| (-4) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
|4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:17|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:17|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 2
2012-10-12 13:42:22|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
|4763|blockrequest| Response class = RPC::XML::fault
|4763|blockrequest| Response type = fault
|4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
|4763|blockrequest| Response value = HASH(0x2a79cf8)
|4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
|4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
|4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-3) vcld, make_new_child (line: 568)
|4763|blockrequest| (-4) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
|4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:22|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:22|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 3
2012-10-12 13:42:27|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
|4763|blockrequest| Response class = RPC::XML::fault
|4763|blockrequest| Response type = fault
|4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
|4763|blockrequest| Response value = HASH(0x294e7a0)
|4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
|4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
|4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-3) vcld, make_new_child (line: 568)
|4763|blockrequest| (-4) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
|4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:27|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:27|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 4
2012-10-12 13:42:32|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
|4763|blockrequest| Response class = RPC::XML::fault
|4763|blockrequest| Response type = fault
|4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
|4763|blockrequest| Response value = HASH(0x294e6b0)
|4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
|4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
|4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-3) vcld, make_new_child (line: 568)
|4763|blockrequest| (-4) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
|4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:32|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:32|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 11 pass 5
2012-10-12 13:42:37|4763|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 11 1
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|utils.pm:xmlrpc_call(9131)|fault occured:
|4763|blockrequest| Response class = RPC::XML::fault
|4763|blockrequest| Response type = fault
|4763|blockrequest| Response string = <fault><value><struct><member><name>faultString</name><value><string>Access denied</string></value></member><member><name>faultCode</name><value><int>3</int></value></member></struct></value></fault>
|4763|blockrequest| Response value = HASH(0x28976f0)
|4763|blockrequest| ( 0) utils.pm, xmlrpc_call (line: 9131)
|4763|blockrequest| (-1) blockrequest.pm, process_block_time (line: 373)
|4763|blockrequest| (-2) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-3) vcld, make_new_child (line: 568)
|4763|blockrequest| (-4) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process_block_time(389)|return argument XMLRPCprocessBlockTime was not a STRUCT as expectedRPC::XML::fault
|4763|blockrequest| ( 0) blockrequest.pm, process_block_time (line: 389)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 193)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $warningmsg in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
|4763|blockrequest| ---- WARNING ----
|4763|blockrequest| 2012-10-12 13:42:37|4763|blockrequest|vcld:warning_handler(610)|Use of uninitialized value $unallocated in concatenation (.) or string at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 212.
|4763|blockrequest| ( 0) vcld, warning_handler (line: 610)
|4763|blockrequest| (-1) blockrequest.pm, process (line: 212)
|4763|blockrequest| (-2) vcld, make_new_child (line: 568)
|4763|blockrequest| (-3) vcld, main (line: 448)
2012-10-12 13:42:37|4763|blockrequest|blockrequest.pm:process(212)|xmlrpc warning: allocated= 0 unallocated=
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
I looked through the RPC::XML::Client perl module code, and in certain cases the send_request subroutine will return a string on failure, though it appears that the VCL code is expecting the response to always be an object. In those circumstances, the $response->type call will fail. There is clearly some underlying error involved, but I think it is being obscured by the error you are seeing.
What I would recommend is that you temporarily add these lines of code just before line 9121:
use Data::Dumper;
notify($ERRORS{'DEBUG'}, 0, "Testing RPC::XML response: " . Dumper($response));
This will stringify the entire $response object (whatever it actually is) so that you can inspect it. See if that yields anything. It should, at least, give you some clues about what is going on.
Aaron
--
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
acoburn@amherst.edu<ma...@amherst.edu>
On Oct 11, 2012, at 12:40 PM, Michael Jinks <mj...@uchicago.edu>> wrote:
I finally got around to trying this; remarks in-line below.
On Tue, Oct 09, 2012 at 08:45:03AM -0400, Aaron Peeler wrote:
I'm not completely sure of the next steps, but I think this has to do
with verifying the hostname of the webserver through the
LWP::UserAgent.
We can try some different options..
One is to go ahead and upgrade to 2.3(what I recommend because of the
improved features)
Not to put too fine a point on it, but... Given the trouble we've had
with our deployment effort up to now, we're extremely reluctant to start
our implementation over again with a new version. Once we have
something that's working in production, sure, we can start a fresh
effort on a development server with the new code, but this effort is
months behind schedule as it is and I really don't want to roll the
dice right now.
So I chose the less intrusive option:
Or another option is to update utils.pm's xmlrpc_call
1st make a copy of your untouched utils.pm file
cp /usr/local/vcl-2.2.1/lib/VCL/utils.pm
/usr/local/vcl-2.2.1/lib/VCL/utils.pm_orig
at line 9114 comment out of utils.pm
The line Should look like
my $cli = RPC::XML::Client->new($XMLRPC_URL);
and replace it with this line:
my $cli = RPC::XML::Client->new($XMLRPC_URL, useragent => ['ssl_opts'
=> {verify_hostname => 0}]);
save and restart vcld process
Hopefully this will help.
The difference in the updating utils.pm option is in the xmlrpc_call
routine is the
After applying this change, I tried again to schedule a block
allocation, and got similar results. Longer log excerpt below, but the
line that jumps out at me is still this one:
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
Since my line 9121 might not match up with the stock code any more,
here's the referenced block:
if ($response->type =~ /fault/){
notify($ERRORS{'WARNING'}, 0, "fault occured:\n" .
" Response class = ".(ref $response)."\n".
" Response type = ".$response->type."\n".
" Response string = ".$response->as_string."\n".
" Response value = ".$response->value."\n"
);
}
...and here's the longer chunk of vcld.log. Any more ideas about what
might be going on? Thanks, -m
2012-10-11 11:34:06|15179|blockrequest|utils.pm:rename_vcld_process(7136)|renamed process to 'vcld 15:20 'michael test''
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:initialize(94)|returning 1
2012-10-11 11:34:06|15179|blockrequest|vcld:make_new_child(565)|VCL::blockrequest object created and initialized
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(158)|blockrequest id: 15
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(159)|blockrequest mode: start
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(160)|blockrequest image id: 114
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(161)|blockrequest number machines: 10
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(162)|blockrequest expire: 2012-10-11 23:59:59
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(163)|blocktime id: 20
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(164)|blocktime processed: 0
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(165)|blocktime start: 2012-10-11 13:00:00
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(166)|owner email: vcl-system@lists.uchicago.edu<ma...@lists.uchicago.edu>
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(167)|help address: vcl-team@lists.uchicago.edu<ma...@lists.uchicago.edu>
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 20
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 20 pass 1
2012-10-11 11:34:06|15179|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 20 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 20, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(20) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x2754050)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x262d8d0)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-11 11:34:06|15179|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 2754050
2012-10-11 11:34:06|15179|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-11 11:34:06|15050|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 15179, signal: CHLD
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
I finally got around to trying this; remarks in-line below.
On Tue, Oct 09, 2012 at 08:45:03AM -0400, Aaron Peeler wrote:
>
> I'm not completely sure of the next steps, but I think this has to do
> with verifying the hostname of the webserver through the
> LWP::UserAgent.
>
> We can try some different options..
>
> One is to go ahead and upgrade to 2.3(what I recommend because of the
> improved features)
Not to put too fine a point on it, but... Given the trouble we've had
with our deployment effort up to now, we're extremely reluctant to start
our implementation over again with a new version. Once we have
something that's working in production, sure, we can start a fresh
effort on a development server with the new code, but this effort is
months behind schedule as it is and I really don't want to roll the
dice right now.
So I chose the less intrusive option:
> Or another option is to update utils.pm's xmlrpc_call
> 1st make a copy of your untouched utils.pm file
> cp /usr/local/vcl-2.2.1/lib/VCL/utils.pm
> /usr/local/vcl-2.2.1/lib/VCL/utils.pm_orig
>
> at line 9114 comment out of utils.pm
>
> The line Should look like
> my $cli = RPC::XML::Client->new($XMLRPC_URL);
>
> and replace it with this line:
>
> my $cli = RPC::XML::Client->new($XMLRPC_URL, useragent => ['ssl_opts'
> => {verify_hostname => 0}]);
>
> save and restart vcld process
>
> Hopefully this will help.
>
> The difference in the updating utils.pm option is in the xmlrpc_call
> routine is the
After applying this change, I tried again to schedule a block
allocation, and got similar results. Longer log excerpt below, but the
line that jumps out at me is still this one:
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
Since my line 9121 might not match up with the stock code any more,
here's the referenced block:
if ($response->type =~ /fault/){
notify($ERRORS{'WARNING'}, 0, "fault occured:\n" .
" Response class = ".(ref $response)."\n".
" Response type = ".$response->type."\n".
" Response string = ".$response->as_string."\n".
" Response value = ".$response->value."\n"
);
}
...and here's the longer chunk of vcld.log. Any more ideas about what
might be going on? Thanks, -m
2012-10-11 11:34:06|15179|blockrequest|utils.pm:rename_vcld_process(7136)|renamed process to 'vcld 15:20 'michael test''
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:initialize(94)|returning 1
2012-10-11 11:34:06|15179|blockrequest|vcld:make_new_child(565)|VCL::blockrequest object created and initialized
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(158)|blockrequest id: 15
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(159)|blockrequest mode: start
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(160)|blockrequest image id: 114
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(161)|blockrequest number machines: 10
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(162)|blockrequest expire: 2012-10-11 23:59:59
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(163)|blocktime id: 20
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(164)|blocktime processed: 0
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(165)|blocktime start: 2012-10-11 13:00:00
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(166)|owner email: vcl-system@lists.uchicago.edu
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(167)|help address: vcl-team@lists.uchicago.edu
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 20
2012-10-11 11:34:06|15179|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 20 pass 1
2012-10-11 11:34:06|15179|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 20 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Found" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Found"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 20, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(20) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x2754050)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x262d8d0)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-11 11:34:06|15179|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 2754050
2012-10-11 11:34:06|15179|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-11 11:34:06|15050|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 15179, signal: CHLD
Re: Block allocations always fail
Posted by Aaron Peeler <aa...@ncsu.edu>.
Hi Michael,
I'm not completely sure of the next steps, but I think this has to do
with verifying the hostname of the webserver through the
LWP::UserAgent.
We can try some different options..
One is to go ahead and upgrade to 2.3(what I recommend because of the
improved features)
https://cwiki.apache.org/confluence/display/VCL/Upgrade+From+Previous+Version+%282.2.1+to+2.3%29
Or another option is to update utils.pm's xmlrpc_call
1st make a copy of your untouched utils.pm file
cp /usr/local/vcl-2.2.1/lib/VCL/utils.pm
/usr/local/vcl-2.2.1/lib/VCL/utils.pm_orig
at line 9114 comment out of utils.pm
The line Should look like
my $cli = RPC::XML::Client->new($XMLRPC_URL);
and replace it with this line:
my $cli = RPC::XML::Client->new($XMLRPC_URL, useragent => ['ssl_opts'
=> {verify_hostname => 0}]);
save and restart vcld process
Hopefully this will help.
The difference in the updating utils.pm option is in the xmlrpc_call
routine is the
Upgrading to 2.3 will have faster load times and improved features.
Aaron
On Mon, Oct 8, 2012 at 6:44 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> Bumping this, with a summary: After fixing a few configuration errors,
> I'm still running into what looks like a cert error when vclclientd
> tries to schedule a block reservation. I'll paste a longer log excerpt
> below in case it's useful, but the salient line seems to be:
>
> HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#2)
>
>
> I've verified that RPC::XML is insalled and working, server time is
> correct. My current hypothesis is that the error message does indeed
> arise from a certificate chain problem, but I can't figure out how to
> test that, nor do I know what vcld is using for a trusted certificate
> store.
>
> I've tried reading through the Perl module stack, down to
> IO/Socket/INET.pm, which I *think* is originating the exception, but
> there my Perl chops fail me and I can't find any place where an SSL
> handshake takes place.
>
> So, an anybody fill in some of the gaps for me? Does it seem like I'm
> on the right track? If this is a cert trust issue, where would I
> install our CA's public key chain so that vcld will use it?
>
> Thanks,
> --Michael
>
>
> Longer vcld log excerpt:
>
> 2012-10-08 15:13:51|28877|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 2 pass 1
> 2012-10-08 15:13:51|28877|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 2 1
> Can't locate object method "type" via package "RPC::XML::Client::send_request:
> HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#2)
> Uncaught exception from user code:
> Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
> at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
> VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 2, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
> VCL::blockrequest::process_block_time(2) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
> VCL::blockrequest::process('VCL::blockrequest=HASH(0x32e6ad0)') called at /usr/local/vcl/bin/vcld line 568
> VCL::vcld::make_new_child('HASH(0x31e54a0)') called at /usr/local/vcl/bin/vcld line 448
> VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
> 2012-10-08 15:13:51|28877|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 32e6ad0
> 2012-10-08 15:13:51|28877|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
> 2012-10-08 15:13:51|5776|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 28877, signal: CHLD
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Bumping this, with a summary: After fixing a few configuration errors,
I'm still running into what looks like a cert error when vclclientd
tries to schedule a block reservation. I'll paste a longer log excerpt
below in case it's useful, but the salient line seems to be:
HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#2)
I've verified that RPC::XML is insalled and working, server time is
correct. My current hypothesis is that the error message does indeed
arise from a certificate chain problem, but I can't figure out how to
test that, nor do I know what vcld is using for a trusted certificate
store.
I've tried reading through the Perl module stack, down to
IO/Socket/INET.pm, which I *think* is originating the exception, but
there my Perl chops fail me and I can't find any place where an SSL
handshake takes place.
So, an anybody fill in some of the gaps for me? Does it seem like I'm
on the right track? If this is a cert trust issue, where would I
install our CA's public key chain so that vcld will use it?
Thanks,
--Michael
Longer vcld log excerpt:
2012-10-08 15:13:51|28877|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 2 pass 1
2012-10-08 15:13:51|28877|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 2 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#2)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab-a.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 2, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(2) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x32e6ad0)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x31e54a0)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-08 15:13:51|28877|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 32e6ad0
2012-10-08 15:13:51|28877|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-08 15:13:51|5776|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 28877, signal: CHLD
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
I've just tried this...
> * Make sure time on management node is correct.
Time is correct.
> * double check the blockRequest table for the given allocation, make
> sure processing is set to 0
It was set to 1, but setting it back to 0 triggered lots of traffic in
the log, see below.
> * make sure the related BlockTimes processing entry in the BlockTimes
> table is set to 0
That was 0.
On the way I uncovered a few more error conditions, involving Perl
libraries that I'd overlooked before, which are now adjusted to call our
purpose-build perl. But I'm back to the error I was getting before:
[...]
2012-10-04 13:21:53|21007|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 19 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 19, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(19) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x102e528)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x208fe70)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-04 13:21:53|21007|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 102e528
2012-10-04 13:21:53|21007|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-04 13:21:53|14005|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 21007, signal: CHLD
So maybe we do have a cert issue. Could it be that the CA bundle that
VCL uses (wherever that is) doesn't like Comodo/AddTrust?
On Thu, Oct 04, 2012 at 11:21:57AM -0400, Aaron Peeler wrote:
> You shouldn't need to wait 45 minutes, the start time only needs to be
> between 6hrs and 30 minutes from now. So as long as the start time is
> between this window, you should see something immediately.
>
> Some things to check:
> * Make sure time on management node is correct.
> * double check the blockRequest table for the given allocation, make
> sure processing is set to 0
> * make sure the related BlockTimes processing entry in the BlockTimes
> table is set to 0
>
> Note, resetting these to 0 will re-processing the blockallocation to
> help debugging a particular block.
>
> -A
>
> On Thu, Oct 4, 2012 at 11:04 AM, Michael Jinks <mj...@uchicago.edu> wrote:
> > Hrm. Well, I made my block reservation, waited my 45 minutes, and still
> > no mention of it in the log. I'll try again with a longer lead time.
> >
> >
> > On Thu, Oct 04, 2012 at 09:01:34AM -0500, Michael Jinks wrote:
> >> Thanks Aaron.
> >>
> >> It looks like we're in good shape as far as this stuff is concerned...
> >>
> >> $ /usr/local/vcl/bin/perl -MRPC::XML -e 'print "success\n";'
> >> success
> >>
> >> $ /usr/local/vcl/bin/perl -MLWP -e 'print LWP->VERSION ."\n";'
> >> 6.04
> >>
> >> ...but I wonder if it's possible that under some circumstances vcld is
> >> calling a perl other than the one we purpose-installed for it. vcld
> >> itself was adjusted to call that instance but a couple of other perl
> >> scripts in vcl/bin weren't, so I've fixed that and scheduled another
> >> reservation 45 minutes from now, will see if that makes any difference.
> >>
> >>
> >>
> >> On Thu, Oct 04, 2012 at 01:42:47PM +0000, Aaron Coburn wrote:
> >> > I have seen something like this before.
> >> >
> >> > I would recommend ensuring that the RPC::XML perl module is properly
> >> > installed on the management node. Try this command:
> >> >
> >> > perl -MRPC::XML -e 'print "success\n";'
> >> >
> >> > If you get errors from this, try manually installing the RPC::XML
> >> > module from here:
> >> >
> >> > [1]http://search.cpan.org/~rjray/RPC-XML-0.77/
> >> >
> >> > The only thing to note is that the current version of RPC::XML requires
> >> > at least version 5.834 of LWP. To check which version of LWP you
> >> > currently have installed, run this command:
> >> >
> >> > perl -MLWP -e 'print LWP->VERSION ."\n";'
> >> >
> >> > If this reports any version before 5.834 (as my system does), then you
> >> > may need to use version 0.72 of the RPC::XML module, which requires
> >> > only version 5.801 of LWP.
> >> >
> >> > Also, when I say 'manually install', I do not mean using the cpan
> >> > interactive interface; rather, download the package with wget, and then
> >> > use the standard perl installation sequence:
> >> >
> >> > perl Makefile.PL
> >> >
> >> > make
> >> >
> >> > make test
> >> >
> >> > make install
> >> >
> >> > Aaron
> >> >
> >> > --
> >> > Aaron Coburn
> >> > Systems Administrator and Programmer
> >> > Academic Technology Services, Amherst College
> >> > [2]acoburn@amherst.edu
> >> > On Oct 4, 2012, at 9:05 AM, Michael Jinks <[3...@uchicago.edu>
> >> > wrote:
> >> >
> >> > Okay, this makes sense. Thanks for clearing that up; I'd wondered
> >> > if
> >> > vcld had logic for ramping up to a reservation.
> >> > I made a reservation last night, scheduled for 08:00 this morning.
> >> > At
> >> > 02:00, I have errors in my log, which I'll paste below. I haven't
> >> > started trying to investigate what's wrong; the error suggests a bad
> >> > cert. Our cert is signed by Comodo, maybe that's the problem? I
> >> > don't
> >> > know what vcld uses as a trust chain.
> >> > There could be several other things going on too. The reference to
> >> > 'Can't locate object method "type"' looks ugly, but I wonder if it's
> >> > just a result of the SSL error further up?
> >> > Log excerpt follows.
> >> > [...]
> >> > 2012-10-04
> >> > 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing
> >> > blocktime_id= 15 pass 1
> >> > 2012-10-04
> >> > 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_strin
> >> > g= XMLRPCprocessBlockTime 15 1
> >> > Can't locate object method "type" via package
> >> > "RPC::XML::Client::send_request:
> >> > HTTP server error: Can't connect to [4]vlab.uchicago.edu:443
> >> > (certificate verify failed)" (perhaps you forgot to load
> >> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> >> > [5]vlab.uchicago.edu:443 (certificate verify failed)"?) at
> >> > /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
> >> > Uncaught exception from user code:
> >> > Can't locate object method "type" via package
> >> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> >> > [6]vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you
> >> > forgot to load "RPC::XML::Client::send_request: HTTP server error:
> >> > Can't connect to [7]vlab.uchicago.edu:443 (certificate verify
> >> > failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line
> >> > 9121.
> >> > at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
> >> > VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1)
> >> > called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line
> >> > 373
> >> > VCL::blockrequest::process_block_time(15) called at
> >> > /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
> >> > VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)
> >> > ') called at /usr/local/vcl/bin/vcld line 568
> >> > VCL::vcld::make_new_child('HASH(0x34740f0)') called at
> >> > /usr/local/vcl/bin/vcld line 448
> >> > VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
> >> > 2012-10-04
> >> > 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest
> >> > destructor called, address: 358bb68
> >> > 2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number
> >> > of database handles state process created: 1
> >> > 2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for
> >> > reservation <unknown>, PID: 3322, signal: CHLD
> >> > On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
> >> >
> >> > Hello Michael,
> >> > The start time for block allocations need to be requested > 30
> >> > minutes
> >> > in advance.
> >> > If folks think this should be adjusted let us know. The original
> >> > thinking was to allow the enough time for the machines to get
> >> > prepared
> >> > for 30+ machines, which is what we've experienced for the average
> >> > number of machines for Blocks allocations.
> >> > Aaron
> >> > On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks
> >> > <[8...@uchicago.edu> wrote:
> >> >
> >> > Thanks, Aaron and Josh.
> >> > I did have some problems in my vcld.conf; the vclsystem account
> >> > hadn't
> >> > been properly configured. So that was clearly part of my trouble.
> >> > Having fixed that, though, I'm still not getting a successful block
> >> > reservation. Based on Josh's remarks about vcld reserving the slots
> >> > "several hours" in advance, I wonder if that's an issue. In order
> >> > to
> >> > test this in a reasonable amount of time, I've been creating a time
> >> > slot
> >> > a few minutes in the future, which then comes and goes with no
> >> > activity
> >> > in the vcld log except 'lastcheckin' lines. I also find, in the
> >> > database:
> >> > mysql> select processed from blockTimes;
> >> > +-----------+
> >> > | processed |
> >> > +-----------+
> >> > | 0 |
> >> > | 0 |
> >> > +-----------+
> >> > 2 rows in set (0.00 sec)
> >> > How far advance do I need to do a block allocation in order for it
> >> > to
> >> > take effect?
> >> > On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
> >> >
> >> > -----BEGIN PGP SIGNED MESSAGE-----
> >> > Hash: SHA1
> >> > Michael,
> >> > Aaron gave some good info.
> >> > One thing I'll point out is that when you view the status of a block
> >> > allocation through the web interface as you've described, the number
> >> > listed for "Failed" comes from
> >> > (number of computers for block allocation) - (number of computers in
> >> > blockComputers table)
> >> > It doesn't necessarily reflect that VCL attempted to load the
> >> > machines
> >> > and that those loads failed. If vcld never processes the block
> >> > allocation, it will always show the full count in the Failed row.
> >> > When the block allocation is created, an entry is created in the
> >> > blockTimes table for each time slot in which the block allocation
> >> > will
> >> > occur. However, no computers are actually allocated for each time
> >> > slot until several hours before a time slot starts. Several hours
> >> > before any block time, vcld processes that block time to allocate
> >> > the
> >> > computers and populate the blockComputers table for that blockTimes
> >> > entry. It does this by making an XMLRPC call to the web code. You
> >> > can check that vcld has actually processed a block time entry by
> >> > looking at the blockTimes.processed field in the database for a
> >> > given
> >> > time slot.
> >> > Josh
> >> > On 10/03/12 13:59, Aaron Coburn wrote:
> >> >
> >> > Michael,
> >> > You should make sure that /etc/vcl/vcld.conf has the proper values
> >> > set for:
> >> > xmlrpc_username xmlrpc_pass xmlrpc_url
> >> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> >> > be something like:
> >> > [9]https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> >> > This is all described in the vcld.conf file as well as on this
> >> > page:
> >> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> >> > ode+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
> >> > Then be sure to set the password for the user identified above
> >> > (e.g. vclsystem@Local). Instructions are here:
> >> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> >> > ode+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemacc
> >> > ountpasswordforxmlrpcapi
> >> > (The instructions are for version 2.3, but they will work for
> >> > 2.2.1 as well)
> >> > Finally, restart vcld so that it can pick up the new information.
> >> > If that doesn't help, then try inspecting the vcld logfile -- it
> >> > will tell you more about why the reservations fail.
> >> > Aaron
> >> > -- Aaron Coburn Systems Administrator and Programmer Academic
> >> > Technology Services, Amherst College acoburn@amherst.edu
> >> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> >> > wrote:
> >> >
> >> > Hi List.
> >> > We have a VCL 2.2 instance running, most things work, but one
> >> > glaring problem is still thwarting me. When I try to create a
> >> > block allocation, the computers always come up "Failed"; this in
> >> > spite of the fact that I can reserve the same image individually
> >> > with no issues.
> >> > I've tried this so far with the admin@Local account, as well as
> >> > my user account which has full admin privileges (or at least as
> >> > close as we can get by assigning rights in the Privileges page).
> >> > I go to the "Block Allocations" page, click "Create New Block
> >> > Allocation", fill in the form, choosing a known-working image in
> >> > the "Environment" dropdown, and a "List of Dates/Times" with one
> >> > time span in the near future. "Submit New Block Allocation" pops
> >> > up a confirmation window, that goes fine, and I have an entry
> >> > "Block Allocations" page under "Manage Block Allocations". Looks
> >> > fine.
> >> > But, if I click an entry under "Your Active Block Allocations",
> >> > under "Current status of computers," the number by "Failed" is
> >> > equal to the number of seats I reserved, and as far as I can tell
> >> > the system never tries to deploy a VM.
> >> > I'm guessing that there is some permission setting or some other
> >> > item within the management node that I've overlooked. Any hints
> >> > on where to look?
> >> > Thanks, -j
> >> > -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
> >> > Services
> >> >
> >> > - --
> >> > - -------------------------------
> >> > Josh Thompson
> >> > VCL Developer
> >> > North Carolina State University
> >> > my GPG/PGP key can be found at [10]pgp.mit.edu
> >> > All electronic mail messages in connection with State business which
> >> > are sent to or received by this account are subject to the NC Public
> >> > Records Law and may be disclosed to third parties.
> >> > -----BEGIN PGP SIGNATURE-----
> >> > Version: GnuPG v2.0.17 (GNU/Linux)
> >> > iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
> >> > SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
> >> > =gnM2
> >> > -----END PGP SIGNATURE-----
> >> >
> >> > --
> >> > Michael Jinks :: [11]mjinks@uchicago.edu :: 773-469-9688
> >> > University of Chicago IT Services
> >> >
> >> > --
> >> > Aaron Peeler
> >> > Program Manager
> >> > Virtual Computing Lab
> >> > NC State University
> >> > All electronic mail messages in connection with State business which
> >> > are sent to or received by this account are subject to the NC Public
> >> > Records Law and may be disclosed to third parties.
> >> >
> >> > --
> >> > Michael Jinks :: [12]mjinks@uchicago.edu :: 773-469-9688
> >> > University of Chicago IT Services
> >> >
> >> > References
> >> >
> >> > 1. http://search.cpan.org/~rjray/RPC-XML-0.77/
> >> > 2. mailto:acoburn@amherst.edu
> >> > 3. mailto:mjinks@uchicago.edu
> >> > 4. http://vlab.uchicago.edu/
> >> > 5. http://vlab.uchicago.edu/
> >> > 6. http://vlab.uchicago.edu/
> >> > 7. http://vlab.uchicago.edu/
> >> > 8. mailto:mjinks@uchicago.edu
> >> > 9. https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> >> > 10. http://pgp.mit.edu/
> >> > 11. mailto:mjinks@uchicago.edu
> >> > 12. mailto:mjinks@uchicago.edu
> >>
> >> --
> >> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> >> University of Chicago IT Services
> >
> > --
> > Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> > University of Chicago IT Services
>
>
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Aaron Peeler <aa...@ncsu.edu>.
You shouldn't need to wait 45 minutes, the start time only needs to be
between 6hrs and 30 minutes from now. So as long as the start time is
between this window, you should see something immediately.
Some things to check:
* Make sure time on management node is correct.
* double check the blockRequest table for the given allocation, make
sure processing is set to 0
* make sure the related BlockTimes processing entry in the BlockTimes
table is set to 0
Note, resetting these to 0 will re-processing the blockallocation to
help debugging a particular block.
-A
On Thu, Oct 4, 2012 at 11:04 AM, Michael Jinks <mj...@uchicago.edu> wrote:
> Hrm. Well, I made my block reservation, waited my 45 minutes, and still
> no mention of it in the log. I'll try again with a longer lead time.
>
>
> On Thu, Oct 04, 2012 at 09:01:34AM -0500, Michael Jinks wrote:
>> Thanks Aaron.
>>
>> It looks like we're in good shape as far as this stuff is concerned...
>>
>> $ /usr/local/vcl/bin/perl -MRPC::XML -e 'print "success\n";'
>> success
>>
>> $ /usr/local/vcl/bin/perl -MLWP -e 'print LWP->VERSION ."\n";'
>> 6.04
>>
>> ...but I wonder if it's possible that under some circumstances vcld is
>> calling a perl other than the one we purpose-installed for it. vcld
>> itself was adjusted to call that instance but a couple of other perl
>> scripts in vcl/bin weren't, so I've fixed that and scheduled another
>> reservation 45 minutes from now, will see if that makes any difference.
>>
>>
>>
>> On Thu, Oct 04, 2012 at 01:42:47PM +0000, Aaron Coburn wrote:
>> > I have seen something like this before.
>> >
>> > I would recommend ensuring that the RPC::XML perl module is properly
>> > installed on the management node. Try this command:
>> >
>> > perl -MRPC::XML -e 'print "success\n";'
>> >
>> > If you get errors from this, try manually installing the RPC::XML
>> > module from here:
>> >
>> > [1]http://search.cpan.org/~rjray/RPC-XML-0.77/
>> >
>> > The only thing to note is that the current version of RPC::XML requires
>> > at least version 5.834 of LWP. To check which version of LWP you
>> > currently have installed, run this command:
>> >
>> > perl -MLWP -e 'print LWP->VERSION ."\n";'
>> >
>> > If this reports any version before 5.834 (as my system does), then you
>> > may need to use version 0.72 of the RPC::XML module, which requires
>> > only version 5.801 of LWP.
>> >
>> > Also, when I say 'manually install', I do not mean using the cpan
>> > interactive interface; rather, download the package with wget, and then
>> > use the standard perl installation sequence:
>> >
>> > perl Makefile.PL
>> >
>> > make
>> >
>> > make test
>> >
>> > make install
>> >
>> > Aaron
>> >
>> > --
>> > Aaron Coburn
>> > Systems Administrator and Programmer
>> > Academic Technology Services, Amherst College
>> > [2]acoburn@amherst.edu
>> > On Oct 4, 2012, at 9:05 AM, Michael Jinks <[3...@uchicago.edu>
>> > wrote:
>> >
>> > Okay, this makes sense. Thanks for clearing that up; I'd wondered
>> > if
>> > vcld had logic for ramping up to a reservation.
>> > I made a reservation last night, scheduled for 08:00 this morning.
>> > At
>> > 02:00, I have errors in my log, which I'll paste below. I haven't
>> > started trying to investigate what's wrong; the error suggests a bad
>> > cert. Our cert is signed by Comodo, maybe that's the problem? I
>> > don't
>> > know what vcld uses as a trust chain.
>> > There could be several other things going on too. The reference to
>> > 'Can't locate object method "type"' looks ugly, but I wonder if it's
>> > just a result of the SSL error further up?
>> > Log excerpt follows.
>> > [...]
>> > 2012-10-04
>> > 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing
>> > blocktime_id= 15 pass 1
>> > 2012-10-04
>> > 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_strin
>> > g= XMLRPCprocessBlockTime 15 1
>> > Can't locate object method "type" via package
>> > "RPC::XML::Client::send_request:
>> > HTTP server error: Can't connect to [4]vlab.uchicago.edu:443
>> > (certificate verify failed)" (perhaps you forgot to load
>> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
>> > [5]vlab.uchicago.edu:443 (certificate verify failed)"?) at
>> > /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
>> > Uncaught exception from user code:
>> > Can't locate object method "type" via package
>> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
>> > [6]vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you
>> > forgot to load "RPC::XML::Client::send_request: HTTP server error:
>> > Can't connect to [7]vlab.uchicago.edu:443 (certificate verify
>> > failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line
>> > 9121.
>> > at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
>> > VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1)
>> > called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line
>> > 373
>> > VCL::blockrequest::process_block_time(15) called at
>> > /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
>> > VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)
>> > ') called at /usr/local/vcl/bin/vcld line 568
>> > VCL::vcld::make_new_child('HASH(0x34740f0)') called at
>> > /usr/local/vcl/bin/vcld line 448
>> > VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
>> > 2012-10-04
>> > 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest
>> > destructor called, address: 358bb68
>> > 2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number
>> > of database handles state process created: 1
>> > 2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for
>> > reservation <unknown>, PID: 3322, signal: CHLD
>> > On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
>> >
>> > Hello Michael,
>> > The start time for block allocations need to be requested > 30
>> > minutes
>> > in advance.
>> > If folks think this should be adjusted let us know. The original
>> > thinking was to allow the enough time for the machines to get
>> > prepared
>> > for 30+ machines, which is what we've experienced for the average
>> > number of machines for Blocks allocations.
>> > Aaron
>> > On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks
>> > <[8...@uchicago.edu> wrote:
>> >
>> > Thanks, Aaron and Josh.
>> > I did have some problems in my vcld.conf; the vclsystem account
>> > hadn't
>> > been properly configured. So that was clearly part of my trouble.
>> > Having fixed that, though, I'm still not getting a successful block
>> > reservation. Based on Josh's remarks about vcld reserving the slots
>> > "several hours" in advance, I wonder if that's an issue. In order
>> > to
>> > test this in a reasonable amount of time, I've been creating a time
>> > slot
>> > a few minutes in the future, which then comes and goes with no
>> > activity
>> > in the vcld log except 'lastcheckin' lines. I also find, in the
>> > database:
>> > mysql> select processed from blockTimes;
>> > +-----------+
>> > | processed |
>> > +-----------+
>> > | 0 |
>> > | 0 |
>> > +-----------+
>> > 2 rows in set (0.00 sec)
>> > How far advance do I need to do a block allocation in order for it
>> > to
>> > take effect?
>> > On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
>> >
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> > Michael,
>> > Aaron gave some good info.
>> > One thing I'll point out is that when you view the status of a block
>> > allocation through the web interface as you've described, the number
>> > listed for "Failed" comes from
>> > (number of computers for block allocation) - (number of computers in
>> > blockComputers table)
>> > It doesn't necessarily reflect that VCL attempted to load the
>> > machines
>> > and that those loads failed. If vcld never processes the block
>> > allocation, it will always show the full count in the Failed row.
>> > When the block allocation is created, an entry is created in the
>> > blockTimes table for each time slot in which the block allocation
>> > will
>> > occur. However, no computers are actually allocated for each time
>> > slot until several hours before a time slot starts. Several hours
>> > before any block time, vcld processes that block time to allocate
>> > the
>> > computers and populate the blockComputers table for that blockTimes
>> > entry. It does this by making an XMLRPC call to the web code. You
>> > can check that vcld has actually processed a block time entry by
>> > looking at the blockTimes.processed field in the database for a
>> > given
>> > time slot.
>> > Josh
>> > On 10/03/12 13:59, Aaron Coburn wrote:
>> >
>> > Michael,
>> > You should make sure that /etc/vcl/vcld.conf has the proper values
>> > set for:
>> > xmlrpc_username xmlrpc_pass xmlrpc_url
>> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
>> > be something like:
>> > [9]https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
>> > This is all described in the vcld.conf file as well as on this
>> > page:
>> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
>> > ode+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
>> > Then be sure to set the password for the user identified above
>> > (e.g. vclsystem@Local). Instructions are here:
>> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
>> > ode+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemacc
>> > ountpasswordforxmlrpcapi
>> > (The instructions are for version 2.3, but they will work for
>> > 2.2.1 as well)
>> > Finally, restart vcld so that it can pick up the new information.
>> > If that doesn't help, then try inspecting the vcld logfile -- it
>> > will tell you more about why the reservations fail.
>> > Aaron
>> > -- Aaron Coburn Systems Administrator and Programmer Academic
>> > Technology Services, Amherst College acoburn@amherst.edu
>> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
>> > wrote:
>> >
>> > Hi List.
>> > We have a VCL 2.2 instance running, most things work, but one
>> > glaring problem is still thwarting me. When I try to create a
>> > block allocation, the computers always come up "Failed"; this in
>> > spite of the fact that I can reserve the same image individually
>> > with no issues.
>> > I've tried this so far with the admin@Local account, as well as
>> > my user account which has full admin privileges (or at least as
>> > close as we can get by assigning rights in the Privileges page).
>> > I go to the "Block Allocations" page, click "Create New Block
>> > Allocation", fill in the form, choosing a known-working image in
>> > the "Environment" dropdown, and a "List of Dates/Times" with one
>> > time span in the near future. "Submit New Block Allocation" pops
>> > up a confirmation window, that goes fine, and I have an entry
>> > "Block Allocations" page under "Manage Block Allocations". Looks
>> > fine.
>> > But, if I click an entry under "Your Active Block Allocations",
>> > under "Current status of computers," the number by "Failed" is
>> > equal to the number of seats I reserved, and as far as I can tell
>> > the system never tries to deploy a VM.
>> > I'm guessing that there is some permission setting or some other
>> > item within the management node that I've overlooked. Any hints
>> > on where to look?
>> > Thanks, -j
>> > -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
>> > Services
>> >
>> > - --
>> > - -------------------------------
>> > Josh Thompson
>> > VCL Developer
>> > North Carolina State University
>> > my GPG/PGP key can be found at [10]pgp.mit.edu
>> > All electronic mail messages in connection with State business which
>> > are sent to or received by this account are subject to the NC Public
>> > Records Law and may be disclosed to third parties.
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v2.0.17 (GNU/Linux)
>> > iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
>> > SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
>> > =gnM2
>> > -----END PGP SIGNATURE-----
>> >
>> > --
>> > Michael Jinks :: [11]mjinks@uchicago.edu :: 773-469-9688
>> > University of Chicago IT Services
>> >
>> > --
>> > Aaron Peeler
>> > Program Manager
>> > Virtual Computing Lab
>> > NC State University
>> > All electronic mail messages in connection with State business which
>> > are sent to or received by this account are subject to the NC Public
>> > Records Law and may be disclosed to third parties.
>> >
>> > --
>> > Michael Jinks :: [12]mjinks@uchicago.edu :: 773-469-9688
>> > University of Chicago IT Services
>> >
>> > References
>> >
>> > 1. http://search.cpan.org/~rjray/RPC-XML-0.77/
>> > 2. mailto:acoburn@amherst.edu
>> > 3. mailto:mjinks@uchicago.edu
>> > 4. http://vlab.uchicago.edu/
>> > 5. http://vlab.uchicago.edu/
>> > 6. http://vlab.uchicago.edu/
>> > 7. http://vlab.uchicago.edu/
>> > 8. mailto:mjinks@uchicago.edu
>> > 9. https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
>> > 10. http://pgp.mit.edu/
>> > 11. mailto:mjinks@uchicago.edu
>> > 12. mailto:mjinks@uchicago.edu
>>
>> --
>> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
>> University of Chicago IT Services
>
> --
> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Hrm. Well, I made my block reservation, waited my 45 minutes, and still
no mention of it in the log. I'll try again with a longer lead time.
On Thu, Oct 04, 2012 at 09:01:34AM -0500, Michael Jinks wrote:
> Thanks Aaron.
>
> It looks like we're in good shape as far as this stuff is concerned...
>
> $ /usr/local/vcl/bin/perl -MRPC::XML -e 'print "success\n";'
> success
>
> $ /usr/local/vcl/bin/perl -MLWP -e 'print LWP->VERSION ."\n";'
> 6.04
>
> ...but I wonder if it's possible that under some circumstances vcld is
> calling a perl other than the one we purpose-installed for it. vcld
> itself was adjusted to call that instance but a couple of other perl
> scripts in vcl/bin weren't, so I've fixed that and scheduled another
> reservation 45 minutes from now, will see if that makes any difference.
>
>
>
> On Thu, Oct 04, 2012 at 01:42:47PM +0000, Aaron Coburn wrote:
> > I have seen something like this before.
> >
> > I would recommend ensuring that the RPC::XML perl module is properly
> > installed on the management node. Try this command:
> >
> > perl -MRPC::XML -e 'print "success\n";'
> >
> > If you get errors from this, try manually installing the RPC::XML
> > module from here:
> >
> > [1]http://search.cpan.org/~rjray/RPC-XML-0.77/
> >
> > The only thing to note is that the current version of RPC::XML requires
> > at least version 5.834 of LWP. To check which version of LWP you
> > currently have installed, run this command:
> >
> > perl -MLWP -e 'print LWP->VERSION ."\n";'
> >
> > If this reports any version before 5.834 (as my system does), then you
> > may need to use version 0.72 of the RPC::XML module, which requires
> > only version 5.801 of LWP.
> >
> > Also, when I say 'manually install', I do not mean using the cpan
> > interactive interface; rather, download the package with wget, and then
> > use the standard perl installation sequence:
> >
> > perl Makefile.PL
> >
> > make
> >
> > make test
> >
> > make install
> >
> > Aaron
> >
> > --
> > Aaron Coburn
> > Systems Administrator and Programmer
> > Academic Technology Services, Amherst College
> > [2]acoburn@amherst.edu
> > On Oct 4, 2012, at 9:05 AM, Michael Jinks <[3...@uchicago.edu>
> > wrote:
> >
> > Okay, this makes sense. Thanks for clearing that up; I'd wondered
> > if
> > vcld had logic for ramping up to a reservation.
> > I made a reservation last night, scheduled for 08:00 this morning.
> > At
> > 02:00, I have errors in my log, which I'll paste below. I haven't
> > started trying to investigate what's wrong; the error suggests a bad
> > cert. Our cert is signed by Comodo, maybe that's the problem? I
> > don't
> > know what vcld uses as a trust chain.
> > There could be several other things going on too. The reference to
> > 'Can't locate object method "type"' looks ugly, but I wonder if it's
> > just a result of the SSL error further up?
> > Log excerpt follows.
> > [...]
> > 2012-10-04
> > 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing
> > blocktime_id= 15 pass 1
> > 2012-10-04
> > 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_strin
> > g= XMLRPCprocessBlockTime 15 1
> > Can't locate object method "type" via package
> > "RPC::XML::Client::send_request:
> > HTTP server error: Can't connect to [4]vlab.uchicago.edu:443
> > (certificate verify failed)" (perhaps you forgot to load
> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> > [5]vlab.uchicago.edu:443 (certificate verify failed)"?) at
> > /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
> > Uncaught exception from user code:
> > Can't locate object method "type" via package
> > "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> > [6]vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you
> > forgot to load "RPC::XML::Client::send_request: HTTP server error:
> > Can't connect to [7]vlab.uchicago.edu:443 (certificate verify
> > failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line
> > 9121.
> > at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
> > VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1)
> > called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line
> > 373
> > VCL::blockrequest::process_block_time(15) called at
> > /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
> > VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)
> > ') called at /usr/local/vcl/bin/vcld line 568
> > VCL::vcld::make_new_child('HASH(0x34740f0)') called at
> > /usr/local/vcl/bin/vcld line 448
> > VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
> > 2012-10-04
> > 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest
> > destructor called, address: 358bb68
> > 2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number
> > of database handles state process created: 1
> > 2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for
> > reservation <unknown>, PID: 3322, signal: CHLD
> > On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
> >
> > Hello Michael,
> > The start time for block allocations need to be requested > 30
> > minutes
> > in advance.
> > If folks think this should be adjusted let us know. The original
> > thinking was to allow the enough time for the machines to get
> > prepared
> > for 30+ machines, which is what we've experienced for the average
> > number of machines for Blocks allocations.
> > Aaron
> > On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks
> > <[8...@uchicago.edu> wrote:
> >
> > Thanks, Aaron and Josh.
> > I did have some problems in my vcld.conf; the vclsystem account
> > hadn't
> > been properly configured. So that was clearly part of my trouble.
> > Having fixed that, though, I'm still not getting a successful block
> > reservation. Based on Josh's remarks about vcld reserving the slots
> > "several hours" in advance, I wonder if that's an issue. In order
> > to
> > test this in a reasonable amount of time, I've been creating a time
> > slot
> > a few minutes in the future, which then comes and goes with no
> > activity
> > in the vcld log except 'lastcheckin' lines. I also find, in the
> > database:
> > mysql> select processed from blockTimes;
> > +-----------+
> > | processed |
> > +-----------+
> > | 0 |
> > | 0 |
> > +-----------+
> > 2 rows in set (0.00 sec)
> > How far advance do I need to do a block allocation in order for it
> > to
> > take effect?
> > On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > Michael,
> > Aaron gave some good info.
> > One thing I'll point out is that when you view the status of a block
> > allocation through the web interface as you've described, the number
> > listed for "Failed" comes from
> > (number of computers for block allocation) - (number of computers in
> > blockComputers table)
> > It doesn't necessarily reflect that VCL attempted to load the
> > machines
> > and that those loads failed. If vcld never processes the block
> > allocation, it will always show the full count in the Failed row.
> > When the block allocation is created, an entry is created in the
> > blockTimes table for each time slot in which the block allocation
> > will
> > occur. However, no computers are actually allocated for each time
> > slot until several hours before a time slot starts. Several hours
> > before any block time, vcld processes that block time to allocate
> > the
> > computers and populate the blockComputers table for that blockTimes
> > entry. It does this by making an XMLRPC call to the web code. You
> > can check that vcld has actually processed a block time entry by
> > looking at the blockTimes.processed field in the database for a
> > given
> > time slot.
> > Josh
> > On 10/03/12 13:59, Aaron Coburn wrote:
> >
> > Michael,
> > You should make sure that /etc/vcl/vcld.conf has the proper values
> > set for:
> > xmlrpc_username xmlrpc_pass xmlrpc_url
> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> > be something like:
> > [9]https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> > This is all described in the vcld.conf file as well as on this
> > page:
> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> > ode+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
> > Then be sure to set the password for the user identified above
> > (e.g. vclsystem@Local). Instructions are here:
> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> > ode+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemacc
> > ountpasswordforxmlrpcapi
> > (The instructions are for version 2.3, but they will work for
> > 2.2.1 as well)
> > Finally, restart vcld so that it can pick up the new information.
> > If that doesn't help, then try inspecting the vcld logfile -- it
> > will tell you more about why the reservations fail.
> > Aaron
> > -- Aaron Coburn Systems Administrator and Programmer Academic
> > Technology Services, Amherst College acoburn@amherst.edu
> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> > wrote:
> >
> > Hi List.
> > We have a VCL 2.2 instance running, most things work, but one
> > glaring problem is still thwarting me. When I try to create a
> > block allocation, the computers always come up "Failed"; this in
> > spite of the fact that I can reserve the same image individually
> > with no issues.
> > I've tried this so far with the admin@Local account, as well as
> > my user account which has full admin privileges (or at least as
> > close as we can get by assigning rights in the Privileges page).
> > I go to the "Block Allocations" page, click "Create New Block
> > Allocation", fill in the form, choosing a known-working image in
> > the "Environment" dropdown, and a "List of Dates/Times" with one
> > time span in the near future. "Submit New Block Allocation" pops
> > up a confirmation window, that goes fine, and I have an entry
> > "Block Allocations" page under "Manage Block Allocations". Looks
> > fine.
> > But, if I click an entry under "Your Active Block Allocations",
> > under "Current status of computers," the number by "Failed" is
> > equal to the number of seats I reserved, and as far as I can tell
> > the system never tries to deploy a VM.
> > I'm guessing that there is some permission setting or some other
> > item within the management node that I've overlooked. Any hints
> > on where to look?
> > Thanks, -j
> > -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
> > Services
> >
> > - --
> > - -------------------------------
> > Josh Thompson
> > VCL Developer
> > North Carolina State University
> > my GPG/PGP key can be found at [10]pgp.mit.edu
> > All electronic mail messages in connection with State business which
> > are sent to or received by this account are subject to the NC Public
> > Records Law and may be disclosed to third parties.
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.17 (GNU/Linux)
> > iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
> > SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
> > =gnM2
> > -----END PGP SIGNATURE-----
> >
> > --
> > Michael Jinks :: [11]mjinks@uchicago.edu :: 773-469-9688
> > University of Chicago IT Services
> >
> > --
> > Aaron Peeler
> > Program Manager
> > Virtual Computing Lab
> > NC State University
> > All electronic mail messages in connection with State business which
> > are sent to or received by this account are subject to the NC Public
> > Records Law and may be disclosed to third parties.
> >
> > --
> > Michael Jinks :: [12]mjinks@uchicago.edu :: 773-469-9688
> > University of Chicago IT Services
> >
> > References
> >
> > 1. http://search.cpan.org/~rjray/RPC-XML-0.77/
> > 2. mailto:acoburn@amherst.edu
> > 3. mailto:mjinks@uchicago.edu
> > 4. http://vlab.uchicago.edu/
> > 5. http://vlab.uchicago.edu/
> > 6. http://vlab.uchicago.edu/
> > 7. http://vlab.uchicago.edu/
> > 8. mailto:mjinks@uchicago.edu
> > 9. https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> > 10. http://pgp.mit.edu/
> > 11. mailto:mjinks@uchicago.edu
> > 12. mailto:mjinks@uchicago.edu
>
> --
> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Thanks Aaron.
It looks like we're in good shape as far as this stuff is concerned...
$ /usr/local/vcl/bin/perl -MRPC::XML -e 'print "success\n";'
success
$ /usr/local/vcl/bin/perl -MLWP -e 'print LWP->VERSION ."\n";'
6.04
...but I wonder if it's possible that under some circumstances vcld is
calling a perl other than the one we purpose-installed for it. vcld
itself was adjusted to call that instance but a couple of other perl
scripts in vcl/bin weren't, so I've fixed that and scheduled another
reservation 45 minutes from now, will see if that makes any difference.
On Thu, Oct 04, 2012 at 01:42:47PM +0000, Aaron Coburn wrote:
> I have seen something like this before.
>
> I would recommend ensuring that the RPC::XML perl module is properly
> installed on the management node. Try this command:
>
> perl -MRPC::XML -e 'print "success\n";'
>
> If you get errors from this, try manually installing the RPC::XML
> module from here:
>
> [1]http://search.cpan.org/~rjray/RPC-XML-0.77/
>
> The only thing to note is that the current version of RPC::XML requires
> at least version 5.834 of LWP. To check which version of LWP you
> currently have installed, run this command:
>
> perl -MLWP -e 'print LWP->VERSION ."\n";'
>
> If this reports any version before 5.834 (as my system does), then you
> may need to use version 0.72 of the RPC::XML module, which requires
> only version 5.801 of LWP.
>
> Also, when I say 'manually install', I do not mean using the cpan
> interactive interface; rather, download the package with wget, and then
> use the standard perl installation sequence:
>
> perl Makefile.PL
>
> make
>
> make test
>
> make install
>
> Aaron
>
> --
> Aaron Coburn
> Systems Administrator and Programmer
> Academic Technology Services, Amherst College
> [2]acoburn@amherst.edu
> On Oct 4, 2012, at 9:05 AM, Michael Jinks <[3...@uchicago.edu>
> wrote:
>
> Okay, this makes sense. Thanks for clearing that up; I'd wondered
> if
> vcld had logic for ramping up to a reservation.
> I made a reservation last night, scheduled for 08:00 this morning.
> At
> 02:00, I have errors in my log, which I'll paste below. I haven't
> started trying to investigate what's wrong; the error suggests a bad
> cert. Our cert is signed by Comodo, maybe that's the problem? I
> don't
> know what vcld uses as a trust chain.
> There could be several other things going on too. The reference to
> 'Can't locate object method "type"' looks ugly, but I wonder if it's
> just a result of the SSL error further up?
> Log excerpt follows.
> [...]
> 2012-10-04
> 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing
> blocktime_id= 15 pass 1
> 2012-10-04
> 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_strin
> g= XMLRPCprocessBlockTime 15 1
> Can't locate object method "type" via package
> "RPC::XML::Client::send_request:
> HTTP server error: Can't connect to [4]vlab.uchicago.edu:443
> (certificate verify failed)" (perhaps you forgot to load
> "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> [5]vlab.uchicago.edu:443 (certificate verify failed)"?) at
> /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
> Uncaught exception from user code:
> Can't locate object method "type" via package
> "RPC::XML::Client::send_request: HTTP server error: Can't connect to
> [6]vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you
> forgot to load "RPC::XML::Client::send_request: HTTP server error:
> Can't connect to [7]vlab.uchicago.edu:443 (certificate verify
> failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line
> 9121.
> at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
> VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1)
> called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line
> 373
> VCL::blockrequest::process_block_time(15) called at
> /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
> VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)
> ') called at /usr/local/vcl/bin/vcld line 568
> VCL::vcld::make_new_child('HASH(0x34740f0)') called at
> /usr/local/vcl/bin/vcld line 448
> VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
> 2012-10-04
> 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest
> destructor called, address: 358bb68
> 2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number
> of database handles state process created: 1
> 2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for
> reservation <unknown>, PID: 3322, signal: CHLD
> On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
>
> Hello Michael,
> The start time for block allocations need to be requested > 30
> minutes
> in advance.
> If folks think this should be adjusted let us know. The original
> thinking was to allow the enough time for the machines to get
> prepared
> for 30+ machines, which is what we've experienced for the average
> number of machines for Blocks allocations.
> Aaron
> On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks
> <[8...@uchicago.edu> wrote:
>
> Thanks, Aaron and Josh.
> I did have some problems in my vcld.conf; the vclsystem account
> hadn't
> been properly configured. So that was clearly part of my trouble.
> Having fixed that, though, I'm still not getting a successful block
> reservation. Based on Josh's remarks about vcld reserving the slots
> "several hours" in advance, I wonder if that's an issue. In order
> to
> test this in a reasonable amount of time, I've been creating a time
> slot
> a few minutes in the future, which then comes and goes with no
> activity
> in the vcld log except 'lastcheckin' lines. I also find, in the
> database:
> mysql> select processed from blockTimes;
> +-----------+
> | processed |
> +-----------+
> | 0 |
> | 0 |
> +-----------+
> 2 rows in set (0.00 sec)
> How far advance do I need to do a block allocation in order for it
> to
> take effect?
> On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Michael,
> Aaron gave some good info.
> One thing I'll point out is that when you view the status of a block
> allocation through the web interface as you've described, the number
> listed for "Failed" comes from
> (number of computers for block allocation) - (number of computers in
> blockComputers table)
> It doesn't necessarily reflect that VCL attempted to load the
> machines
> and that those loads failed. If vcld never processes the block
> allocation, it will always show the full count in the Failed row.
> When the block allocation is created, an entry is created in the
> blockTimes table for each time slot in which the block allocation
> will
> occur. However, no computers are actually allocated for each time
> slot until several hours before a time slot starts. Several hours
> before any block time, vcld processes that block time to allocate
> the
> computers and populate the blockComputers table for that blockTimes
> entry. It does this by making an XMLRPC call to the web code. You
> can check that vcld has actually processed a block time entry by
> looking at the blockTimes.processed field in the database for a
> given
> time slot.
> Josh
> On 10/03/12 13:59, Aaron Coburn wrote:
>
> Michael,
> You should make sure that /etc/vcl/vcld.conf has the proper values
> set for:
> xmlrpc_username xmlrpc_pass xmlrpc_url
> xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> be something like:
> [9]https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> This is all described in the vcld.conf file as well as on this
> page:
> https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> ode+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
> Then be sure to set the password for the user identified above
> (e.g. vclsystem@Local). Instructions are here:
> https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+N
> ode+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemacc
> ountpasswordforxmlrpcapi
> (The instructions are for version 2.3, but they will work for
> 2.2.1 as well)
> Finally, restart vcld so that it can pick up the new information.
> If that doesn't help, then try inspecting the vcld logfile -- it
> will tell you more about why the reservations fail.
> Aaron
> -- Aaron Coburn Systems Administrator and Programmer Academic
> Technology Services, Amherst College acoburn@amherst.edu
> On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> wrote:
>
> Hi List.
> We have a VCL 2.2 instance running, most things work, but one
> glaring problem is still thwarting me. When I try to create a
> block allocation, the computers always come up "Failed"; this in
> spite of the fact that I can reserve the same image individually
> with no issues.
> I've tried this so far with the admin@Local account, as well as
> my user account which has full admin privileges (or at least as
> close as we can get by assigning rights in the Privileges page).
> I go to the "Block Allocations" page, click "Create New Block
> Allocation", fill in the form, choosing a known-working image in
> the "Environment" dropdown, and a "List of Dates/Times" with one
> time span in the near future. "Submit New Block Allocation" pops
> up a confirmation window, that goes fine, and I have an entry
> "Block Allocations" page under "Manage Block Allocations". Looks
> fine.
> But, if I click an entry under "Your Active Block Allocations",
> under "Current status of computers," the number by "Failed" is
> equal to the number of seats I reserved, and as far as I can tell
> the system never tries to deploy a VM.
> I'm guessing that there is some permission setting or some other
> item within the management node that I've overlooked. Any hints
> on where to look?
> Thanks, -j
> -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
> Services
>
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
> my GPG/PGP key can be found at [10]pgp.mit.edu
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
> SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
> =gnM2
> -----END PGP SIGNATURE-----
>
> --
> Michael Jinks :: [11]mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
>
> --
> Michael Jinks :: [12]mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
>
> References
>
> 1. http://search.cpan.org/~rjray/RPC-XML-0.77/
> 2. mailto:acoburn@amherst.edu
> 3. mailto:mjinks@uchicago.edu
> 4. http://vlab.uchicago.edu/
> 5. http://vlab.uchicago.edu/
> 6. http://vlab.uchicago.edu/
> 7. http://vlab.uchicago.edu/
> 8. mailto:mjinks@uchicago.edu
> 9. https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> 10. http://pgp.mit.edu/
> 11. mailto:mjinks@uchicago.edu
> 12. mailto:mjinks@uchicago.edu
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
I have seen something like this before.
I would recommend ensuring that the RPC::XML perl module is properly installed on the management node. Try this command:
perl -MRPC::XML -e 'print "success\n";'
If you get errors from this, try manually installing the RPC::XML module from here:
http://search.cpan.org/~rjray/RPC-XML-0.77/
The only thing to note is that the current version of RPC::XML requires at least version 5.834 of LWP. To check which version of LWP you currently have installed, run this command:
perl -MLWP -e 'print LWP->VERSION ."\n";'
If this reports any version before 5.834 (as my system does), then you may need to use version 0.72 of the RPC::XML module, which requires only version 5.801 of LWP.
Also, when I say 'manually install', I do not mean using the cpan interactive interface; rather, download the package with wget, and then use the standard perl installation sequence:
perl Makefile.PL
make
make test
make install
Aaron
--
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
acoburn@amherst.edu<ma...@amherst.edu>
On Oct 4, 2012, at 9:05 AM, Michael Jinks <mj...@uchicago.edu>> wrote:
Okay, this makes sense. Thanks for clearing that up; I'd wondered if
vcld had logic for ramping up to a reservation.
I made a reservation last night, scheduled for 08:00 this morning. At
02:00, I have errors in my log, which I'll paste below. I haven't
started trying to investigate what's wrong; the error suggests a bad
cert. Our cert is signed by Comodo, maybe that's the problem? I don't
know what vcld uses as a trust chain.
There could be several other things going on too. The reference to
'Can't locate object method "type"' looks ugly, but I wonder if it's
just a result of the SSL error further up?
Log excerpt follows.
[...]
2012-10-04 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 15 pass 1
2012-10-04 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 15 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Can't connect to vlab.uchicago.edu<http://vlab.uchicago.edu>:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu<http://vlab.uchicago.edu>:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu<http://vlab.uchicago.edu>:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu<http://vlab.uchicago.edu>:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(15) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x34740f0)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 358bb68
2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 3322, signal: CHLD
On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
Hello Michael,
The start time for block allocations need to be requested > 30 minutes
in advance.
If folks think this should be adjusted let us know. The original
thinking was to allow the enough time for the machines to get prepared
for 30+ machines, which is what we've experienced for the average
number of machines for Blocks allocations.
Aaron
On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks <mj...@uchicago.edu>> wrote:
Thanks, Aaron and Josh.
I did have some problems in my vcld.conf; the vclsystem account hadn't
been properly configured. So that was clearly part of my trouble.
Having fixed that, though, I'm still not getting a successful block
reservation. Based on Josh's remarks about vcld reserving the slots
"several hours" in advance, I wonder if that's an issue. In order to
test this in a reasonable amount of time, I've been creating a time slot
a few minutes in the future, which then comes and goes with no activity
in the vcld log except 'lastcheckin' lines. I also find, in the
database:
mysql> select processed from blockTimes;
+-----------+
| processed |
+-----------+
| 0 |
| 0 |
+-----------+
2 rows in set (0.00 sec)
How far advance do I need to do a block allocation in order for it to
take effect?
On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael,
Aaron gave some good info.
One thing I'll point out is that when you view the status of a block
allocation through the web interface as you've described, the number
listed for "Failed" comes from
(number of computers for block allocation) - (number of computers in
blockComputers table)
It doesn't necessarily reflect that VCL attempted to load the machines
and that those loads failed. If vcld never processes the block
allocation, it will always show the full count in the Failed row.
When the block allocation is created, an entry is created in the
blockTimes table for each time slot in which the block allocation will
occur. However, no computers are actually allocated for each time
slot until several hours before a time slot starts. Several hours
before any block time, vcld processes that block time to allocate the
computers and populate the blockComputers table for that blockTimes
entry. It does this by making an XMLRPC call to the web code. You
can check that vcld has actually processed a block time entry by
looking at the blockTimes.processed field in the database for a given
time slot.
Josh
On 10/03/12 13:59, Aaron Coburn wrote:
Michael,
You should make sure that /etc/vcl/vcld.conf has the proper values
set for:
xmlrpc_username xmlrpc_pass xmlrpc_url
xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
be something like:
https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
This is all described in the vcld.conf file as well as on this
page:
https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
Then be sure to set the password for the user identified above
(e.g. vclsystem@Local). Instructions are here:
https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
(The instructions are for version 2.3, but they will work for
2.2.1 as well)
Finally, restart vcld so that it can pick up the new information.
If that doesn't help, then try inspecting the vcld logfile -- it
will tell you more about why the reservations fail.
Aaron
-- Aaron Coburn Systems Administrator and Programmer Academic
Technology Services, Amherst College acoburn@amherst.edu
On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
wrote:
Hi List.
We have a VCL 2.2 instance running, most things work, but one
glaring problem is still thwarting me. When I try to create a
block allocation, the computers always come up "Failed"; this in
spite of the fact that I can reserve the same image individually
with no issues.
I've tried this so far with the admin@Local account, as well as
my user account which has full admin privileges (or at least as
close as we can get by assigning rights in the Privileges page).
I go to the "Block Allocations" page, click "Create New Block
Allocation", fill in the form, choosing a known-working image in
the "Environment" dropdown, and a "List of Dates/Times" with one
time span in the near future. "Submit New Block Allocation" pops
up a confirmation window, that goes fine, and I have an entry
"Block Allocations" page under "Manage Block Allocations". Looks
fine.
But, if I click an entry under "Your Active Block Allocations",
under "Current status of computers," the number by "Failed" is
equal to the number of seats I reserved, and as far as I can tell
the system never tries to deploy a VM.
I'm guessing that there is some permission setting or some other
item within the management node that I've overlooked. Any hints
on where to look?
Thanks, -j
-- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
Services
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu<http://pgp.mit.edu>
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
=gnM2
-----END PGP SIGNATURE-----
--
Michael Jinks :: mjinks@uchicago.edu<ma...@uchicago.edu> :: 773-469-9688
University of Chicago IT Services
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
--
Michael Jinks :: mjinks@uchicago.edu<ma...@uchicago.edu> :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Okay, this makes sense. Thanks for clearing that up; I'd wondered if
vcld had logic for ramping up to a reservation.
I made a reservation last night, scheduled for 08:00 this morning. At
02:00, I have errors in my log, which I'll paste below. I haven't
started trying to investigate what's wrong; the error suggests a bad
cert. Our cert is signed by Comodo, maybe that's the problem? I don't
know what vcld uses as a trust chain.
There could be several other things going on too. The reference to
'Can't locate object method "type"' looks ugly, but I wonder if it's
just a result of the SSL error further up?
Log excerpt follows.
[...]
2012-10-04 02:00:02|3322|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 15 pass 1
2012-10-04 02:00:02|3322|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 15 1
Can't locate object method "type" via package "RPC::XML::Client::send_request:
HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121 (#1)
Uncaught exception from user code:
Can't locate object method "type" via package "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)" (perhaps you forgot to load "RPC::XML::Client::send_request: HTTP server error: Can't connect to vlab.uchicago.edu:443 (certificate verify failed)"?) at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121.
at /usr/local/vcl-2.2.1/bin/../lib/VCL/utils.pm line 9121
VCL::utils::xmlrpc_call('XMLRPCprocessBlockTime', 15, 1) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 373
VCL::blockrequest::process_block_time(15) called at /usr/local/vcl-2.2.1/bin/../lib/VCL/blockrequest.pm line 193
VCL::blockrequest::process('VCL::blockrequest=HASH(0x358bb68)') called at /usr/local/vcl/bin/vcld line 568
VCL::vcld::make_new_child('HASH(0x34740f0)') called at /usr/local/vcl/bin/vcld line 448
VCL::vcld::main() called at /usr/local/vcl/bin/vcld line 98
2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(829)|VCL::blockrequest destructor called, address: 358bb68
2012-10-04 02:00:02|3322|blockrequest|State.pm:DESTROY(848)|number of database handles state process created: 1
2012-10-04 02:00:02|20597|vcld:REAPER(718)|VCL process exited for reservation <unknown>, PID: 3322, signal: CHLD
On Thu, Oct 04, 2012 at 08:48:34AM -0400, Aaron Peeler wrote:
> Hello Michael,
>
> The start time for block allocations need to be requested > 30 minutes
> in advance.
>
> If folks think this should be adjusted let us know. The original
> thinking was to allow the enough time for the machines to get prepared
> for 30+ machines, which is what we've experienced for the average
> number of machines for Blocks allocations.
>
> Aaron
>
> On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> > Thanks, Aaron and Josh.
> >
> > I did have some problems in my vcld.conf; the vclsystem account hadn't
> > been properly configured. So that was clearly part of my trouble.
> >
> > Having fixed that, though, I'm still not getting a successful block
> > reservation. Based on Josh's remarks about vcld reserving the slots
> > "several hours" in advance, I wonder if that's an issue. In order to
> > test this in a reasonable amount of time, I've been creating a time slot
> > a few minutes in the future, which then comes and goes with no activity
> > in the vcld log except 'lastcheckin' lines. I also find, in the
> > database:
> >
> > mysql> select processed from blockTimes;
> > +-----------+
> > | processed |
> > +-----------+
> > | 0 |
> > | 0 |
> > +-----------+
> > 2 rows in set (0.00 sec)
> >
> >
> > How far advance do I need to do a block allocation in order for it to
> > take effect?
> >
> >
> >
> > On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Michael,
> >>
> >> Aaron gave some good info.
> >>
> >> One thing I'll point out is that when you view the status of a block
> >> allocation through the web interface as you've described, the number
> >> listed for "Failed" comes from
> >>
> >> (number of computers for block allocation) - (number of computers in
> >> blockComputers table)
> >>
> >> It doesn't necessarily reflect that VCL attempted to load the machines
> >> and that those loads failed. If vcld never processes the block
> >> allocation, it will always show the full count in the Failed row.
> >>
> >> When the block allocation is created, an entry is created in the
> >> blockTimes table for each time slot in which the block allocation will
> >> occur. However, no computers are actually allocated for each time
> >> slot until several hours before a time slot starts. Several hours
> >> before any block time, vcld processes that block time to allocate the
> >> computers and populate the blockComputers table for that blockTimes
> >> entry. It does this by making an XMLRPC call to the web code. You
> >> can check that vcld has actually processed a block time entry by
> >> looking at the blockTimes.processed field in the database for a given
> >> time slot.
> >>
> >> Josh
> >>
> >> On 10/03/12 13:59, Aaron Coburn wrote:
> >> > Michael,
> >> >
> >> > You should make sure that /etc/vcl/vcld.conf has the proper values
> >> > set for:
> >> >
> >> > xmlrpc_username xmlrpc_pass xmlrpc_url
> >> >
> >> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> >> > be something like:
> >> > https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> >> >
> >> > This is all described in the vcld.conf file as well as on this
> >> > page:
> >> >
> >> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
> >> >
> >> > Then be sure to set the password for the user identified above
> >> > (e.g. vclsystem@Local). Instructions are here:
> >> >
> >> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
> >> >
> >> > (The instructions are for version 2.3, but they will work for
> >> > 2.2.1 as well)
> >> >
> >> > Finally, restart vcld so that it can pick up the new information.
> >> >
> >> > If that doesn't help, then try inspecting the vcld logfile -- it
> >> > will tell you more about why the reservations fail.
> >> >
> >> > Aaron
> >> >
> >> >
> >> >
> >> > -- Aaron Coburn Systems Administrator and Programmer Academic
> >> > Technology Services, Amherst College acoburn@amherst.edu
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> >> > wrote:
> >> >
> >> >> Hi List.
> >> >>
> >> >> We have a VCL 2.2 instance running, most things work, but one
> >> >> glaring problem is still thwarting me. When I try to create a
> >> >> block allocation, the computers always come up "Failed"; this in
> >> >> spite of the fact that I can reserve the same image individually
> >> >> with no issues.
> >> >>
> >> >> I've tried this so far with the admin@Local account, as well as
> >> >> my user account which has full admin privileges (or at least as
> >> >> close as we can get by assigning rights in the Privileges page).
> >> >>
> >> >> I go to the "Block Allocations" page, click "Create New Block
> >> >> Allocation", fill in the form, choosing a known-working image in
> >> >> the "Environment" dropdown, and a "List of Dates/Times" with one
> >> >> time span in the near future. "Submit New Block Allocation" pops
> >> >> up a confirmation window, that goes fine, and I have an entry
> >> >> "Block Allocations" page under "Manage Block Allocations". Looks
> >> >> fine.
> >> >>
> >> >> But, if I click an entry under "Your Active Block Allocations",
> >> >> under "Current status of computers," the number by "Failed" is
> >> >> equal to the number of seats I reserved, and as far as I can tell
> >> >> the system never tries to deploy a VM.
> >> >>
> >> >> I'm guessing that there is some permission setting or some other
> >> >> item within the management node that I've overlooked. Any hints
> >> >> on where to look?
> >> >>
> >> >> Thanks, -j
> >> >>
> >> >> -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
> >> >> Services
> >> >
> >>
> >> - --
> >> - -------------------------------
> >> Josh Thompson
> >> VCL Developer
> >> North Carolina State University
> >>
> >> my GPG/PGP key can be found at pgp.mit.edu
> >>
> >> All electronic mail messages in connection with State business which
> >> are sent to or received by this account are subject to the NC Public
> >> Records Law and may be disclosed to third parties.
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v2.0.17 (GNU/Linux)
> >>
> >> iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
> >> SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
> >> =gnM2
> >> -----END PGP SIGNATURE-----
> >
> > --
> > Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> > University of Chicago IT Services
>
>
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Aaron Peeler <aa...@ncsu.edu>.
Hello Michael,
The start time for block allocations need to be requested > 30 minutes
in advance.
If folks think this should be adjusted let us know. The original
thinking was to allow the enough time for the machines to get prepared
for 30+ machines, which is what we've experienced for the average
number of machines for Blocks allocations.
Aaron
On Wed, Oct 3, 2012 at 6:03 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> Thanks, Aaron and Josh.
>
> I did have some problems in my vcld.conf; the vclsystem account hadn't
> been properly configured. So that was clearly part of my trouble.
>
> Having fixed that, though, I'm still not getting a successful block
> reservation. Based on Josh's remarks about vcld reserving the slots
> "several hours" in advance, I wonder if that's an issue. In order to
> test this in a reasonable amount of time, I've been creating a time slot
> a few minutes in the future, which then comes and goes with no activity
> in the vcld log except 'lastcheckin' lines. I also find, in the
> database:
>
> mysql> select processed from blockTimes;
> +-----------+
> | processed |
> +-----------+
> | 0 |
> | 0 |
> +-----------+
> 2 rows in set (0.00 sec)
>
>
> How far advance do I need to do a block allocation in order for it to
> take effect?
>
>
>
> On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Michael,
>>
>> Aaron gave some good info.
>>
>> One thing I'll point out is that when you view the status of a block
>> allocation through the web interface as you've described, the number
>> listed for "Failed" comes from
>>
>> (number of computers for block allocation) - (number of computers in
>> blockComputers table)
>>
>> It doesn't necessarily reflect that VCL attempted to load the machines
>> and that those loads failed. If vcld never processes the block
>> allocation, it will always show the full count in the Failed row.
>>
>> When the block allocation is created, an entry is created in the
>> blockTimes table for each time slot in which the block allocation will
>> occur. However, no computers are actually allocated for each time
>> slot until several hours before a time slot starts. Several hours
>> before any block time, vcld processes that block time to allocate the
>> computers and populate the blockComputers table for that blockTimes
>> entry. It does this by making an XMLRPC call to the web code. You
>> can check that vcld has actually processed a block time entry by
>> looking at the blockTimes.processed field in the database for a given
>> time slot.
>>
>> Josh
>>
>> On 10/03/12 13:59, Aaron Coburn wrote:
>> > Michael,
>> >
>> > You should make sure that /etc/vcl/vcld.conf has the proper values
>> > set for:
>> >
>> > xmlrpc_username xmlrpc_pass xmlrpc_url
>> >
>> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
>> > be something like:
>> > https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
>> >
>> > This is all described in the vcld.conf file as well as on this
>> > page:
>> >
>> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
>> >
>> > Then be sure to set the password for the user identified above
>> > (e.g. vclsystem@Local). Instructions are here:
>> >
>> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
>> >
>> > (The instructions are for version 2.3, but they will work for
>> > 2.2.1 as well)
>> >
>> > Finally, restart vcld so that it can pick up the new information.
>> >
>> > If that doesn't help, then try inspecting the vcld logfile -- it
>> > will tell you more about why the reservations fail.
>> >
>> > Aaron
>> >
>> >
>> >
>> > -- Aaron Coburn Systems Administrator and Programmer Academic
>> > Technology Services, Amherst College acoburn@amherst.edu
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
>> > wrote:
>> >
>> >> Hi List.
>> >>
>> >> We have a VCL 2.2 instance running, most things work, but one
>> >> glaring problem is still thwarting me. When I try to create a
>> >> block allocation, the computers always come up "Failed"; this in
>> >> spite of the fact that I can reserve the same image individually
>> >> with no issues.
>> >>
>> >> I've tried this so far with the admin@Local account, as well as
>> >> my user account which has full admin privileges (or at least as
>> >> close as we can get by assigning rights in the Privileges page).
>> >>
>> >> I go to the "Block Allocations" page, click "Create New Block
>> >> Allocation", fill in the form, choosing a known-working image in
>> >> the "Environment" dropdown, and a "List of Dates/Times" with one
>> >> time span in the near future. "Submit New Block Allocation" pops
>> >> up a confirmation window, that goes fine, and I have an entry
>> >> "Block Allocations" page under "Manage Block Allocations". Looks
>> >> fine.
>> >>
>> >> But, if I click an entry under "Your Active Block Allocations",
>> >> under "Current status of computers," the number by "Failed" is
>> >> equal to the number of seats I reserved, and as far as I can tell
>> >> the system never tries to deploy a VM.
>> >>
>> >> I'm guessing that there is some permission setting or some other
>> >> item within the management node that I've overlooked. Any hints
>> >> on where to look?
>> >>
>> >> Thanks, -j
>> >>
>> >> -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
>> >> Services
>> >
>>
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.17 (GNU/Linux)
>>
>> iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
>> SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
>> =gnM2
>> -----END PGP SIGNATURE-----
>
> --
> Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
> University of Chicago IT Services
--
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
Thanks, Aaron and Josh.
I did have some problems in my vcld.conf; the vclsystem account hadn't
been properly configured. So that was clearly part of my trouble.
Having fixed that, though, I'm still not getting a successful block
reservation. Based on Josh's remarks about vcld reserving the slots
"several hours" in advance, I wonder if that's an issue. In order to
test this in a reasonable amount of time, I've been creating a time slot
a few minutes in the future, which then comes and goes with no activity
in the vcld log except 'lastcheckin' lines. I also find, in the
database:
mysql> select processed from blockTimes;
+-----------+
| processed |
+-----------+
| 0 |
| 0 |
+-----------+
2 rows in set (0.00 sec)
How far advance do I need to do a block allocation in order for it to
take effect?
On Wed, Oct 03, 2012 at 03:03:46PM -0400, Josh Thompson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michael,
>
> Aaron gave some good info.
>
> One thing I'll point out is that when you view the status of a block
> allocation through the web interface as you've described, the number
> listed for "Failed" comes from
>
> (number of computers for block allocation) - (number of computers in
> blockComputers table)
>
> It doesn't necessarily reflect that VCL attempted to load the machines
> and that those loads failed. If vcld never processes the block
> allocation, it will always show the full count in the Failed row.
>
> When the block allocation is created, an entry is created in the
> blockTimes table for each time slot in which the block allocation will
> occur. However, no computers are actually allocated for each time
> slot until several hours before a time slot starts. Several hours
> before any block time, vcld processes that block time to allocate the
> computers and populate the blockComputers table for that blockTimes
> entry. It does this by making an XMLRPC call to the web code. You
> can check that vcld has actually processed a block time entry by
> looking at the blockTimes.processed field in the database for a given
> time slot.
>
> Josh
>
> On 10/03/12 13:59, Aaron Coburn wrote:
> > Michael,
> >
> > You should make sure that /etc/vcl/vcld.conf has the proper values
> > set for:
> >
> > xmlrpc_username xmlrpc_pass xmlrpc_url
> >
> > xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> > be something like:
> > https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
> >
> > This is all described in the vcld.conf file as well as on this
> > page:
> >
> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
> >
> > Then be sure to set the password for the user identified above
> > (e.g. vclsystem@Local). Instructions are here:
> >
> > https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
> >
> > (The instructions are for version 2.3, but they will work for
> > 2.2.1 as well)
> >
> > Finally, restart vcld so that it can pick up the new information.
> >
> > If that doesn't help, then try inspecting the vcld logfile -- it
> > will tell you more about why the reservations fail.
> >
> > Aaron
> >
> >
> >
> > -- Aaron Coburn Systems Administrator and Programmer Academic
> > Technology Services, Amherst College acoburn@amherst.edu
> >
> >
> >
> >
> >
> >
> > On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> > wrote:
> >
> >> Hi List.
> >>
> >> We have a VCL 2.2 instance running, most things work, but one
> >> glaring problem is still thwarting me. When I try to create a
> >> block allocation, the computers always come up "Failed"; this in
> >> spite of the fact that I can reserve the same image individually
> >> with no issues.
> >>
> >> I've tried this so far with the admin@Local account, as well as
> >> my user account which has full admin privileges (or at least as
> >> close as we can get by assigning rights in the Privileges page).
> >>
> >> I go to the "Block Allocations" page, click "Create New Block
> >> Allocation", fill in the form, choosing a known-working image in
> >> the "Environment" dropdown, and a "List of Dates/Times" with one
> >> time span in the near future. "Submit New Block Allocation" pops
> >> up a confirmation window, that goes fine, and I have an entry
> >> "Block Allocations" page under "Manage Block Allocations". Looks
> >> fine.
> >>
> >> But, if I click an entry under "Your Active Block Allocations",
> >> under "Current status of computers," the number by "Failed" is
> >> equal to the number of seats I reserved, and as far as I can tell
> >> the system never tries to deploy a VM.
> >>
> >> I'm guessing that there is some permission setting or some other
> >> item within the management node that I've overlooked. Any hints
> >> on where to look?
> >>
> >> Thanks, -j
> >>
> >> -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
> >> Services
> >
>
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
>
> iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
> SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
> =gnM2
> -----END PGP SIGNATURE-----
--
Michael Jinks :: mjinks@uchicago.edu :: 773-469-9688
University of Chicago IT Services
Re: Block allocations always fail
Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael,
Aaron gave some good info.
One thing I'll point out is that when you view the status of a block
allocation through the web interface as you've described, the number
listed for "Failed" comes from
(number of computers for block allocation) - (number of computers in
blockComputers table)
It doesn't necessarily reflect that VCL attempted to load the machines
and that those loads failed. If vcld never processes the block
allocation, it will always show the full count in the Failed row.
When the block allocation is created, an entry is created in the
blockTimes table for each time slot in which the block allocation will
occur. However, no computers are actually allocated for each time
slot until several hours before a time slot starts. Several hours
before any block time, vcld processes that block time to allocate the
computers and populate the blockComputers table for that blockTimes
entry. It does this by making an XMLRPC call to the web code. You
can check that vcld has actually processed a block time entry by
looking at the blockTimes.processed field in the database for a given
time slot.
Josh
On 10/03/12 13:59, Aaron Coburn wrote:
> Michael,
>
> You should make sure that /etc/vcl/vcld.conf has the proper values
> set for:
>
> xmlrpc_username xmlrpc_pass xmlrpc_url
>
> xmlrpc_username should be set as vclsystem@Local xmlrpc_url should
> be something like:
> https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
>
> This is all described in the vcld.conf file as well as on this
> page:
>
> https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
>
> Then be sure to set the password for the user identified above
> (e.g. vclsystem@Local). Instructions are here:
>
> https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
>
> (The instructions are for version 2.3, but they will work for
> 2.2.1 as well)
>
> Finally, restart vcld so that it can pick up the new information.
>
> If that doesn't help, then try inspecting the vcld logfile -- it
> will tell you more about why the reservations fail.
>
> Aaron
>
>
>
> -- Aaron Coburn Systems Administrator and Programmer Academic
> Technology Services, Amherst College acoburn@amherst.edu
>
>
>
>
>
>
> On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu>
> wrote:
>
>> Hi List.
>>
>> We have a VCL 2.2 instance running, most things work, but one
>> glaring problem is still thwarting me. When I try to create a
>> block allocation, the computers always come up "Failed"; this in
>> spite of the fact that I can reserve the same image individually
>> with no issues.
>>
>> I've tried this so far with the admin@Local account, as well as
>> my user account which has full admin privileges (or at least as
>> close as we can get by assigning rights in the Privileges page).
>>
>> I go to the "Block Allocations" page, click "Create New Block
>> Allocation", fill in the form, choosing a known-working image in
>> the "Environment" dropdown, and a "List of Dates/Times" with one
>> time span in the near future. "Submit New Block Allocation" pops
>> up a confirmation window, that goes fine, and I have an entry
>> "Block Allocations" page under "Manage Block Allocations". Looks
>> fine.
>>
>> But, if I click an entry under "Your Active Block Allocations",
>> under "Current status of computers," the number by "Failed" is
>> equal to the number of seats I reserved, and as far as I can tell
>> the system never tries to deploy a VM.
>>
>> I'm guessing that there is some permission setting or some other
>> item within the management node that I've overlooked. Any hints
>> on where to look?
>>
>> Thanks, -j
>>
>> -- Michael Jinks :: mjinks@uchicago.edu University of Chicago IT
>> Services
>
- --
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University
my GPG/PGP key can be found at pgp.mit.edu
All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
iEYEARECAAYFAlBsjBIACgkQV/LQcNdtPQOcyQCfTQE1TzGXVFsIqitpPFrzJrM/
SoIAn3p/PMlH+mjI0jWpHdwOC12/hwyu
=gnM2
-----END PGP SIGNATURE-----
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
Michael,
You should make sure that /etc/vcl/vcld.conf has the proper values set for:
xmlrpc_username
xmlrpc_pass
xmlrpc_url
xmlrpc_username should be set as vclsystem@Local
xmlrpc_url should be something like: https://vcl.myinstitution.edu/index.php?mode=xmlrpccall
This is all described in the vcld.conf file as well as on this page:
https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Configurevcld.conf
Then be sure to set the password for the user identified above (e.g. vclsystem@Local). Instructions are here:
https://cwiki.apache.org/confluence/display/VCL/VCL+2.3+Management+Node+Installation#VCL2.3ManagementNodeInstallation-Setthevclsystemaccountpasswordforxmlrpcapi
(The instructions are for version 2.3, but they will work for 2.2.1 as well)
Finally, restart vcld so that it can pick up the new information.
If that doesn't help, then try inspecting the vcld logfile -- it will tell you more about why the reservations fail.
Aaron
--
Aaron Coburn
Systems Administrator and Programmer
Academic Technology Services, Amherst College
acoburn@amherst.edu
On Oct 3, 2012, at 1:30 PM, Michael Jinks <mj...@uchicago.edu> wrote:
> Hi List.
>
> We have a VCL 2.2 instance running, most things work, but one glaring
> problem is still thwarting me. When I try to create a block allocation,
> the computers always come up "Failed"; this in spite of the fact that I
> can reserve the same image individually with no issues.
>
> I've tried this so far with the admin@Local account, as well as my user
> account which has full admin privileges (or at least as close as we can
> get by assigning rights in the Privileges page).
>
> I go to the "Block Allocations" page, click "Create New Block
> Allocation", fill in the form, choosing a known-working image in the
> "Environment" dropdown, and a "List of Dates/Times" with one time span
> in the near future. "Submit New Block Allocation" pops up a
> confirmation window, that goes fine, and I have an entry "Block
> Allocations" page under "Manage Block Allocations". Looks fine.
>
> But, if I click an entry under "Your Active Block Allocations", under
> "Current status of computers," the number by "Failed" is equal to the
> number of seats I reserved, and as far as I can tell the system never
> tries to deploy a VM.
>
> I'm guessing that there is some permission setting or some other item
> within the management node that I've overlooked. Any hints on where to
> look?
>
> Thanks,
> -j
>
> --
> Michael Jinks :: mjinks@uchicago.edu
> University of Chicago IT Services
Re: Block allocations always fail
Posted by Aaron Coburn <ac...@amherst.edu>.
I would recommend looking at this thread:
http://markmail.org/message/cwt33ptwqrbwteh5#query:+page:1+mid:4ywid347fjat4mxl+state:results
In general, I wouldn't recommend deleting system users like this. The default database setup expects the vclsystem account to have id=3, so when you recreate the record in the database, you may get a different ID. At the very least, this modified ID will conflict with the default setting for $xmlrpcBlockAPIUsers in .ht-inc/conf.php. You will at least need to modify the values of that array to include the ID of your new vclsystem user.
By the way, it is possible but there is no interface for changing local account passwords in 2.2.1 (an interface for this was added for 2.3). If you need to change a password, however, this message describes how to do it:
http://mail-archives.apache.org/mod_mbox/vcl-user/201209.mbox/%3C504F39A7.4090807@ncsu.edu%3E
Aaron
On Oct 12, 2012, at 11:37 AM, Michael Jinks <mj...@uchicago.edu>
wrote:
> I'm trying Aaron Coburn's approach to getting better debug information
> from the XML::RPC code in utils.pm. But along the way I've hit a fresh
> snag.
>
> I had been working on (what will be) our production VCL server, but
> since I'm tinkering with the code I moved my effort to the development
> instance. Now, when I try to make a block reservation I get a different
> error. From vcld.log:
>
> 2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(167)|help address:
> 2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 7
> 2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 1
> 2012-10-12 10:19:50|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
> 2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
> 2012-10-12 10:19:55|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 2
> 2012-10-12 10:19:55|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
> 2012-10-12 10:19:55|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
> 2012-10-12 10:20:00|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 3
> 2012-10-12 10:20:00|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
> 2012-10-12 10:20:01|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
>
>
> That repeats for three more cycles and then it gives up.
>
> From the "access denied" error I thought we might have a problem with
> our vclsystem account, but there's no way to change or verify the
> password, so I deleted it (and its foreign key records) from the
> database and recreated it. No difference.
>
> So what's broken now?
>
>
> --
> Michael Jinks :: mjinks@uchicago.edu
> University of Chicago IT Services
Re: Block allocations always fail
Posted by Michael Jinks <mj...@uchicago.edu>.
I'm trying Aaron Coburn's approach to getting better debug information
from the XML::RPC code in utils.pm. But along the way I've hit a fresh
snag.
I had been working on (what will be) our production VCL server, but
since I'm tinkering with the code I moved my effort to the development
instance. Now, when I try to make a block reservation I get a different
error. From vcld.log:
2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(167)|help address:
2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(172)|updated process flag on blocktime_id= 7
2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 1
2012-10-12 10:19:50|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
2012-10-12 10:19:50|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
2012-10-12 10:19:55|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 2
2012-10-12 10:19:55|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
2012-10-12 10:19:55|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
2012-10-12 10:20:00|2477|blockrequest|blockrequest.pm:process(192)|processing blocktime_id= 7 pass 3
2012-10-12 10:20:00|2477|blockrequest|utils.pm:xmlrpc_call(9105)|argument_string= XMLRPCprocessBlockTime 7 1
2012-10-12 10:20:01|2477|blockrequest|blockrequest.pm:process(216)|xmlrpc error on blockrequest_id=6 blocktime_id=7 : access denied for managing block allocations
That repeats for three more cycles and then it gives up.
>From the "access denied" error I thought we might have a problem with
our vclsystem account, but there's no way to change or verify the
password, so I deleted it (and its foreign key records) from the
database and recreated it. No difference.
So what's broken now?
--
Michael Jinks :: mjinks@uchicago.edu
University of Chicago IT Services