You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Darren Shepherd <da...@gmail.com> on 2013/10/31 16:36:08 UTC
Sending to LocalHostEndpoint, Is there any resolution for this?
Edison, Min,
When creating a Zone and there is no SSVM so it sends to
LocalHostEndPoint. I always get the below error. Is there some fix
for this?
ERROR [o.a.c.s.r.NfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) Unable to create local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
ERROR [o.a.c.s.r.LocalNfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) GetRootDir for
nfs://192.168.3.134:/exports/secondary1 failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.ensureLocalRootPathExists(NfsSecondaryStorageResource.java:2456)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.mount(LocalNfsSecondaryStorageResource.java:96)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mountUri(NfsSecondaryStorageResource.java:2293)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.getRootDir(LocalNfsSecondaryStorageResource.java:85)
at org.apache.cloudstack.storage.template.DownloadManagerImpl.handleDownloadCommand(DownloadManagerImpl.java:695)
Darren
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Min Chen <mi...@citrix.com>.
Yes, this issue exist in 4.2/4.2.1 since DefaultEndpointSelector was
introduced in 4.2.
Thanks
-min
On 10/31/13 10:56 AM, "Darren Shepherd" <da...@gmail.com>
wrote:
>Does this issue exist in 4.2/4.2.1? I'm going to put the bug as blocker,
>if you disagree we can lower it. I really think it is quite a bad
>scenario
>to mount NFS to the mgmt server.
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 10:48 AM, Min Chen <mi...@citrix.com> wrote:
>
>> Delaying them until ssvm is up may not that easy to handle in current
>> CloudStack. We may be able to send an error in this case by
>>distinguishing
>> S3 register template case from other scenarios. Would you mind filing a
>> JIRA bug on this? Your issue seems different from the issue filed by
>>Daan
>> (https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
>>
>> Thanks
>> -min
>>
>>
>> On 10/31/13 10:38 AM, "Darren Shepherd" <da...@gmail.com>
>> wrote:
>>
>> >The behaviour should be to delay until there is a SSVM available. I
>>don't
>> >know if that is even possible in CloudStack, I don't want you to
>>sleep().
>> >Is there any way to delay? Something else in the system should be
>> >ensuring
>> >that a SSVM is running and available (what I don't know...). If you
>>can't
>> >delay, it is far better in my mind to send an error to the user than to
>> >try
>> >to do the operation on the mgmt server.
>> >
>> >I think it is very dangerous to have this code that routes to localhost
>> >because in a production setup ACS will have the sudo access to
>> >mount/mkdir. But the server may not have network access, so you run
>>the
>> >chance of mounting the NFS and having it hang almost indefinitely which
>> >can
>> >cause system instability for the mgmt server.
>> >
>> >The specific issue I have right now is that for some reason on master,
>>the
>> >allocator is not working for me anymore. So SSVM doesn't deploy. If I
>> >restart the mgmt stack, on host connect it tries to download templates
>>so
>> >it sends a command and it goes to LocalHostEndPoint.
>> >
>> >Darren
>> >
>> >
>> >On Thu, Oct 31, 2013 at 10:28 AM, Min Chen <mi...@citrix.com> wrote:
>> >
>> >> Yes, that is true, it is a bug in that case. The logic of choosing
>> >> endpoint to send Command will not work properly for
>>destroying/Stopping
>> >>VM
>> >> case. Our assumption is that registering template is done when ssvm
>>is
>> >>up
>> >> and running. Otherwise, it will send to LocalHost to execute
>>command. In
>> >> your opinion, what should be expected behavior if user is trying to
>> >> register template when ssvm is crashed? Directly reporting error
>>instead
>> >> of trying anything.
>> >>
>> >> Thanks
>> >> -min
>> >>
>> >> On 10/31/13 10:18 AM, "Darren Shepherd" <da...@gmail.com>
>> >> wrote:
>> >>
>> >> >It's happening when I first create the zone. I'll debug it further
>>to
>> >> >tell
>> >> >you exactly what its happening. But I don't understand how the
>> >>following
>> >> >code in DefaultEndpointSelector could be correct. If I have a zone,
>> >>and I
>> >> >delete/stop/crash the SSVM, and at the same time register a new
>> >>template,
>> >> >it seems the DownloadCommand will go to LocalHostEndPoint.
>> >> >
>> >> > protected EndPoint findEndpointForImageStorage(DataStore store)
>>{
>> >> > Long dcId = null;
>> >> > Scope storeScope = store.getScope();
>> >> > if (storeScope.getScopeType() == ScopeType.ZONE) {
>> >> > dcId = storeScope.getScopeId();
>> >> > }
>> >> > // find ssvm that can be used to download data to store. For
>> >> >zone-wide
>> >> > // image store, use SSVM for that zone. For region-wide
>>store,
>> >> > // we can arbitrarily pick one ssvm to do that task
>> >> > List<HostVO> ssAHosts =
>> >> >listUpAndConnectingSecondaryStorageVmHost(dcId);
>> >> > if (ssAHosts == null || ssAHosts.isEmpty()) {
>> >> > s_logger.info("No running ssvm is found, so command will
>> be
>> >> >sent to LocalHostEndPoint");
>> >> > return LocalHostEndpoint.getEndpoint(); // use local
>>host
>> >>as
>> >> >endpoint in
>> >> > // case of no ssvm existing
>> >> > }
>> >> > Collections.shuffle(ssAHosts);
>> >> > HostVO host = ssAHosts.get(0);
>> >> > return
>> >>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>> >> >host.getPrivateIpAddress(),
>> >> > host.getPublicIpAddress());
>> >> > }
>> >> >
>> >> >Darren
>> >> >
>> >> >
>> >> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com>
>>wrote:
>> >> >
>> >> >> Hi Darren,
>> >> >>
>> >> >> The logic of sending command to Local in case of no SSVM
>>is
>> >>up
>> >> >>is
>> >> >> to
>> >> >> support S3 as secondary storage. In case of S3 as secondary
>>storage,
>> >> >> system vm template is automatically downloaded to S3 when S3 is
>>added
>> >> >>into
>> >> >> CloudStack, unlike NFS secondary storage case where we used a
>>script
>> >>to
>> >> >> pre-populate system vm template. The thing I don't get is: what
>> >>scenario
>> >> >> are you trying to do to trigger DownloadCommand when SSVM is not
>>up?
>> >> >>
>> >> >> Thanks
>> >> >> -min
>> >> >>
>> >> >> On 10/31/13 9:32 AM, "Darren Shepherd"
>><da...@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >> >No, that's not the problem. The problem is that the
>>DownloadCommand
>> >> >> >is supposed to go to the SSVM, but it sending it to the
>> >> >> >LocalHostEndPoint which is the mgmt server. So the mgmt server
>>can
>> >> >> >not create /var/cloudstack/mnt/
>> >> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt
>> >>setup
>> >> >> >so that it does not have sudo access. A standard cloudstack
>>install
>> >> >> >has sudo access so it will create those folders and mount on the
>> >>mgmt
>> >> >> >server, but it shouldn't.
>> >> >> >
>> >> >> >Please don't tell me we made a change so that the mgmt server is
>> >> >> >expected to have NFS access to all secondary storages.
>> >> >> >
>> >> >> >Darren
>> >> >> >
>> >> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com>
>> >> >>wrote:
>> >> >> >> Please check if you can
>> >> >> >> create
>> >> >>
>> >>>>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> >> >> >> and mount nfs point.
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Darren Shepherd <da...@gmail.com>.
Does this issue exist in 4.2/4.2.1? I'm going to put the bug as blocker,
if you disagree we can lower it. I really think it is quite a bad scenario
to mount NFS to the mgmt server.
Darren
On Thu, Oct 31, 2013 at 10:48 AM, Min Chen <mi...@citrix.com> wrote:
> Delaying them until ssvm is up may not that easy to handle in current
> CloudStack. We may be able to send an error in this case by distinguishing
> S3 register template case from other scenarios. Would you mind filing a
> JIRA bug on this? Your issue seems different from the issue filed by Daan
> (https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
>
> Thanks
> -min
>
>
> On 10/31/13 10:38 AM, "Darren Shepherd" <da...@gmail.com>
> wrote:
>
> >The behaviour should be to delay until there is a SSVM available. I don't
> >know if that is even possible in CloudStack, I don't want you to sleep().
> >Is there any way to delay? Something else in the system should be
> >ensuring
> >that a SSVM is running and available (what I don't know...). If you can't
> >delay, it is far better in my mind to send an error to the user than to
> >try
> >to do the operation on the mgmt server.
> >
> >I think it is very dangerous to have this code that routes to localhost
> >because in a production setup ACS will have the sudo access to
> >mount/mkdir. But the server may not have network access, so you run the
> >chance of mounting the NFS and having it hang almost indefinitely which
> >can
> >cause system instability for the mgmt server.
> >
> >The specific issue I have right now is that for some reason on master, the
> >allocator is not working for me anymore. So SSVM doesn't deploy. If I
> >restart the mgmt stack, on host connect it tries to download templates so
> >it sends a command and it goes to LocalHostEndPoint.
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 10:28 AM, Min Chen <mi...@citrix.com> wrote:
> >
> >> Yes, that is true, it is a bug in that case. The logic of choosing
> >> endpoint to send Command will not work properly for destroying/Stopping
> >>VM
> >> case. Our assumption is that registering template is done when ssvm is
> >>up
> >> and running. Otherwise, it will send to LocalHost to execute command. In
> >> your opinion, what should be expected behavior if user is trying to
> >> register template when ssvm is crashed? Directly reporting error instead
> >> of trying anything.
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 10:18 AM, "Darren Shepherd" <da...@gmail.com>
> >> wrote:
> >>
> >> >It's happening when I first create the zone. I'll debug it further to
> >> >tell
> >> >you exactly what its happening. But I don't understand how the
> >>following
> >> >code in DefaultEndpointSelector could be correct. If I have a zone,
> >>and I
> >> >delete/stop/crash the SSVM, and at the same time register a new
> >>template,
> >> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >> >
> >> > protected EndPoint findEndpointForImageStorage(DataStore store) {
> >> > Long dcId = null;
> >> > Scope storeScope = store.getScope();
> >> > if (storeScope.getScopeType() == ScopeType.ZONE) {
> >> > dcId = storeScope.getScopeId();
> >> > }
> >> > // find ssvm that can be used to download data to store. For
> >> >zone-wide
> >> > // image store, use SSVM for that zone. For region-wide store,
> >> > // we can arbitrarily pick one ssvm to do that task
> >> > List<HostVO> ssAHosts =
> >> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> >> > if (ssAHosts == null || ssAHosts.isEmpty()) {
> >> > s_logger.info("No running ssvm is found, so command will
> be
> >> >sent to LocalHostEndPoint");
> >> > return LocalHostEndpoint.getEndpoint(); // use local host
> >>as
> >> >endpoint in
> >> > // case of no ssvm existing
> >> > }
> >> > Collections.shuffle(ssAHosts);
> >> > HostVO host = ssAHosts.get(0);
> >> > return
> >>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >> >host.getPrivateIpAddress(),
> >> > host.getPublicIpAddress());
> >> > }
> >> >
> >> >Darren
> >> >
> >> >
> >> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com> wrote:
> >> >
> >> >> Hi Darren,
> >> >>
> >> >> The logic of sending command to Local in case of no SSVM is
> >>up
> >> >>is
> >> >> to
> >> >> support S3 as secondary storage. In case of S3 as secondary storage,
> >> >> system vm template is automatically downloaded to S3 when S3 is added
> >> >>into
> >> >> CloudStack, unlike NFS secondary storage case where we used a script
> >>to
> >> >> pre-populate system vm template. The thing I don't get is: what
> >>scenario
> >> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
> >> >>
> >> >> Thanks
> >> >> -min
> >> >>
> >> >> On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com>
> >> >> wrote:
> >> >>
> >> >> >No, that's not the problem. The problem is that the DownloadCommand
> >> >> >is supposed to go to the SSVM, but it sending it to the
> >> >> >LocalHostEndPoint which is the mgmt server. So the mgmt server can
> >> >> >not create /var/cloudstack/mnt/
> >> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt
> >>setup
> >> >> >so that it does not have sudo access. A standard cloudstack install
> >> >> >has sudo access so it will create those folders and mount on the
> >>mgmt
> >> >> >server, but it shouldn't.
> >> >> >
> >> >> >Please don't tell me we made a change so that the mgmt server is
> >> >> >expected to have NFS access to all secondary storages.
> >> >> >
> >> >> >Darren
> >> >> >
> >> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com>
> >> >>wrote:
> >> >> >> Please check if you can
> >> >> >> create
> >> >>
> >>>>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> >> >> and mount nfs point.
> >> >>
> >> >>
> >>
> >>
>
>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Min Chen <mi...@citrix.com>.
Delaying them until ssvm is up may not that easy to handle in current
CloudStack. We may be able to send an error in this case by distinguishing
S3 register template case from other scenarios. Would you mind filing a
JIRA bug on this? Your issue seems different from the issue filed by Daan
(https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
Thanks
-min
On 10/31/13 10:38 AM, "Darren Shepherd" <da...@gmail.com>
wrote:
>The behaviour should be to delay until there is a SSVM available. I don't
>know if that is even possible in CloudStack, I don't want you to sleep().
>Is there any way to delay? Something else in the system should be
>ensuring
>that a SSVM is running and available (what I don't know...). If you can't
>delay, it is far better in my mind to send an error to the user than to
>try
>to do the operation on the mgmt server.
>
>I think it is very dangerous to have this code that routes to localhost
>because in a production setup ACS will have the sudo access to
>mount/mkdir. But the server may not have network access, so you run the
>chance of mounting the NFS and having it hang almost indefinitely which
>can
>cause system instability for the mgmt server.
>
>The specific issue I have right now is that for some reason on master, the
>allocator is not working for me anymore. So SSVM doesn't deploy. If I
>restart the mgmt stack, on host connect it tries to download templates so
>it sends a command and it goes to LocalHostEndPoint.
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 10:28 AM, Min Chen <mi...@citrix.com> wrote:
>
>> Yes, that is true, it is a bug in that case. The logic of choosing
>> endpoint to send Command will not work properly for destroying/Stopping
>>VM
>> case. Our assumption is that registering template is done when ssvm is
>>up
>> and running. Otherwise, it will send to LocalHost to execute command. In
>> your opinion, what should be expected behavior if user is trying to
>> register template when ssvm is crashed? Directly reporting error instead
>> of trying anything.
>>
>> Thanks
>> -min
>>
>> On 10/31/13 10:18 AM, "Darren Shepherd" <da...@gmail.com>
>> wrote:
>>
>> >It's happening when I first create the zone. I'll debug it further to
>> >tell
>> >you exactly what its happening. But I don't understand how the
>>following
>> >code in DefaultEndpointSelector could be correct. If I have a zone,
>>and I
>> >delete/stop/crash the SSVM, and at the same time register a new
>>template,
>> >it seems the DownloadCommand will go to LocalHostEndPoint.
>> >
>> > protected EndPoint findEndpointForImageStorage(DataStore store) {
>> > Long dcId = null;
>> > Scope storeScope = store.getScope();
>> > if (storeScope.getScopeType() == ScopeType.ZONE) {
>> > dcId = storeScope.getScopeId();
>> > }
>> > // find ssvm that can be used to download data to store. For
>> >zone-wide
>> > // image store, use SSVM for that zone. For region-wide store,
>> > // we can arbitrarily pick one ssvm to do that task
>> > List<HostVO> ssAHosts =
>> >listUpAndConnectingSecondaryStorageVmHost(dcId);
>> > if (ssAHosts == null || ssAHosts.isEmpty()) {
>> > s_logger.info("No running ssvm is found, so command will be
>> >sent to LocalHostEndPoint");
>> > return LocalHostEndpoint.getEndpoint(); // use local host
>>as
>> >endpoint in
>> > // case of no ssvm existing
>> > }
>> > Collections.shuffle(ssAHosts);
>> > HostVO host = ssAHosts.get(0);
>> > return
>>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>> >host.getPrivateIpAddress(),
>> > host.getPublicIpAddress());
>> > }
>> >
>> >Darren
>> >
>> >
>> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com> wrote:
>> >
>> >> Hi Darren,
>> >>
>> >> The logic of sending command to Local in case of no SSVM is
>>up
>> >>is
>> >> to
>> >> support S3 as secondary storage. In case of S3 as secondary storage,
>> >> system vm template is automatically downloaded to S3 when S3 is added
>> >>into
>> >> CloudStack, unlike NFS secondary storage case where we used a script
>>to
>> >> pre-populate system vm template. The thing I don't get is: what
>>scenario
>> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
>> >>
>> >> Thanks
>> >> -min
>> >>
>> >> On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com>
>> >> wrote:
>> >>
>> >> >No, that's not the problem. The problem is that the DownloadCommand
>> >> >is supposed to go to the SSVM, but it sending it to the
>> >> >LocalHostEndPoint which is the mgmt server. So the mgmt server can
>> >> >not create /var/cloudstack/mnt/
>> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt
>>setup
>> >> >so that it does not have sudo access. A standard cloudstack install
>> >> >has sudo access so it will create those folders and mount on the
>>mgmt
>> >> >server, but it shouldn't.
>> >> >
>> >> >Please don't tell me we made a change so that the mgmt server is
>> >> >expected to have NFS access to all secondary storages.
>> >> >
>> >> >Darren
>> >> >
>> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com>
>> >>wrote:
>> >> >> Please check if you can
>> >> >> create
>> >>
>>>>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> >> >> and mount nfs point.
>> >>
>> >>
>>
>>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Darren Shepherd <da...@gmail.com>.
The behaviour should be to delay until there is a SSVM available. I don't
know if that is even possible in CloudStack, I don't want you to sleep().
Is there any way to delay? Something else in the system should be ensuring
that a SSVM is running and available (what I don't know...). If you can't
delay, it is far better in my mind to send an error to the user than to try
to do the operation on the mgmt server.
I think it is very dangerous to have this code that routes to localhost
because in a production setup ACS will have the sudo access to
mount/mkdir. But the server may not have network access, so you run the
chance of mounting the NFS and having it hang almost indefinitely which can
cause system instability for the mgmt server.
The specific issue I have right now is that for some reason on master, the
allocator is not working for me anymore. So SSVM doesn't deploy. If I
restart the mgmt stack, on host connect it tries to download templates so
it sends a command and it goes to LocalHostEndPoint.
Darren
On Thu, Oct 31, 2013 at 10:28 AM, Min Chen <mi...@citrix.com> wrote:
> Yes, that is true, it is a bug in that case. The logic of choosing
> endpoint to send Command will not work properly for destroying/Stopping VM
> case. Our assumption is that registering template is done when ssvm is up
> and running. Otherwise, it will send to LocalHost to execute command. In
> your opinion, what should be expected behavior if user is trying to
> register template when ssvm is crashed? Directly reporting error instead
> of trying anything.
>
> Thanks
> -min
>
> On 10/31/13 10:18 AM, "Darren Shepherd" <da...@gmail.com>
> wrote:
>
> >It's happening when I first create the zone. I'll debug it further to
> >tell
> >you exactly what its happening. But I don't understand how the following
> >code in DefaultEndpointSelector could be correct. If I have a zone, and I
> >delete/stop/crash the SSVM, and at the same time register a new template,
> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >
> > protected EndPoint findEndpointForImageStorage(DataStore store) {
> > Long dcId = null;
> > Scope storeScope = store.getScope();
> > if (storeScope.getScopeType() == ScopeType.ZONE) {
> > dcId = storeScope.getScopeId();
> > }
> > // find ssvm that can be used to download data to store. For
> >zone-wide
> > // image store, use SSVM for that zone. For region-wide store,
> > // we can arbitrarily pick one ssvm to do that task
> > List<HostVO> ssAHosts =
> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> > if (ssAHosts == null || ssAHosts.isEmpty()) {
> > s_logger.info("No running ssvm is found, so command will be
> >sent to LocalHostEndPoint");
> > return LocalHostEndpoint.getEndpoint(); // use local host as
> >endpoint in
> > // case of no ssvm existing
> > }
> > Collections.shuffle(ssAHosts);
> > HostVO host = ssAHosts.get(0);
> > return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >host.getPrivateIpAddress(),
> > host.getPublicIpAddress());
> > }
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com> wrote:
> >
> >> Hi Darren,
> >>
> >> The logic of sending command to Local in case of no SSVM is up
> >>is
> >> to
> >> support S3 as secondary storage. In case of S3 as secondary storage,
> >> system vm template is automatically downloaded to S3 when S3 is added
> >>into
> >> CloudStack, unlike NFS secondary storage case where we used a script to
> >> pre-populate system vm template. The thing I don't get is: what scenario
> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com>
> >> wrote:
> >>
> >> >No, that's not the problem. The problem is that the DownloadCommand
> >> >is supposed to go to the SSVM, but it sending it to the
> >> >LocalHostEndPoint which is the mgmt server. So the mgmt server can
> >> >not create /var/cloudstack/mnt/
> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt setup
> >> >so that it does not have sudo access. A standard cloudstack install
> >> >has sudo access so it will create those folders and mount on the mgmt
> >> >server, but it shouldn't.
> >> >
> >> >Please don't tell me we made a change so that the mgmt server is
> >> >expected to have NFS access to all secondary storages.
> >> >
> >> >Darren
> >> >
> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com>
> >>wrote:
> >> >> Please check if you can
> >> >> create
> >> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> >> and mount nfs point.
> >>
> >>
>
>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Min Chen <mi...@citrix.com>.
Yes, that is true, it is a bug in that case. The logic of choosing
endpoint to send Command will not work properly for destroying/Stopping VM
case. Our assumption is that registering template is done when ssvm is up
and running. Otherwise, it will send to LocalHost to execute command. In
your opinion, what should be expected behavior if user is trying to
register template when ssvm is crashed? Directly reporting error instead
of trying anything.
Thanks
-min
On 10/31/13 10:18 AM, "Darren Shepherd" <da...@gmail.com>
wrote:
>It's happening when I first create the zone. I'll debug it further to
>tell
>you exactly what its happening. But I don't understand how the following
>code in DefaultEndpointSelector could be correct. If I have a zone, and I
>delete/stop/crash the SSVM, and at the same time register a new template,
>it seems the DownloadCommand will go to LocalHostEndPoint.
>
> protected EndPoint findEndpointForImageStorage(DataStore store) {
> Long dcId = null;
> Scope storeScope = store.getScope();
> if (storeScope.getScopeType() == ScopeType.ZONE) {
> dcId = storeScope.getScopeId();
> }
> // find ssvm that can be used to download data to store. For
>zone-wide
> // image store, use SSVM for that zone. For region-wide store,
> // we can arbitrarily pick one ssvm to do that task
> List<HostVO> ssAHosts =
>listUpAndConnectingSecondaryStorageVmHost(dcId);
> if (ssAHosts == null || ssAHosts.isEmpty()) {
> s_logger.info("No running ssvm is found, so command will be
>sent to LocalHostEndPoint");
> return LocalHostEndpoint.getEndpoint(); // use local host as
>endpoint in
> // case of no ssvm existing
> }
> Collections.shuffle(ssAHosts);
> HostVO host = ssAHosts.get(0);
> return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>host.getPrivateIpAddress(),
> host.getPublicIpAddress());
> }
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com> wrote:
>
>> Hi Darren,
>>
>> The logic of sending command to Local in case of no SSVM is up
>>is
>> to
>> support S3 as secondary storage. In case of S3 as secondary storage,
>> system vm template is automatically downloaded to S3 when S3 is added
>>into
>> CloudStack, unlike NFS secondary storage case where we used a script to
>> pre-populate system vm template. The thing I don't get is: what scenario
>> are you trying to do to trigger DownloadCommand when SSVM is not up?
>>
>> Thanks
>> -min
>>
>> On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com>
>> wrote:
>>
>> >No, that's not the problem. The problem is that the DownloadCommand
>> >is supposed to go to the SSVM, but it sending it to the
>> >LocalHostEndPoint which is the mgmt server. So the mgmt server can
>> >not create /var/cloudstack/mnt/
>> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt setup
>> >so that it does not have sudo access. A standard cloudstack install
>> >has sudo access so it will create those folders and mount on the mgmt
>> >server, but it shouldn't.
>> >
>> >Please don't tell me we made a change so that the mgmt server is
>> >expected to have NFS access to all secondary storages.
>> >
>> >Darren
>> >
>> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com>
>>wrote:
>> >> Please check if you can
>> >> create
>> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> >> and mount nfs point.
>>
>>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Darren Shepherd <da...@gmail.com>.
It's happening when I first create the zone. I'll debug it further to tell
you exactly what its happening. But I don't understand how the following
code in DefaultEndpointSelector could be correct. If I have a zone, and I
delete/stop/crash the SSVM, and at the same time register a new template,
it seems the DownloadCommand will go to LocalHostEndPoint.
protected EndPoint findEndpointForImageStorage(DataStore store) {
Long dcId = null;
Scope storeScope = store.getScope();
if (storeScope.getScopeType() == ScopeType.ZONE) {
dcId = storeScope.getScopeId();
}
// find ssvm that can be used to download data to store. For
zone-wide
// image store, use SSVM for that zone. For region-wide store,
// we can arbitrarily pick one ssvm to do that task
List<HostVO> ssAHosts =
listUpAndConnectingSecondaryStorageVmHost(dcId);
if (ssAHosts == null || ssAHosts.isEmpty()) {
s_logger.info("No running ssvm is found, so command will be
sent to LocalHostEndPoint");
return LocalHostEndpoint.getEndpoint(); // use local host as
endpoint in
// case of no ssvm existing
}
Collections.shuffle(ssAHosts);
HostVO host = ssAHosts.get(0);
return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
host.getPrivateIpAddress(),
host.getPublicIpAddress());
}
Darren
On Thu, Oct 31, 2013 at 9:42 AM, Min Chen <mi...@citrix.com> wrote:
> Hi Darren,
>
> The logic of sending command to Local in case of no SSVM is up is
> to
> support S3 as secondary storage. In case of S3 as secondary storage,
> system vm template is automatically downloaded to S3 when S3 is added into
> CloudStack, unlike NFS secondary storage case where we used a script to
> pre-populate system vm template. The thing I don't get is: what scenario
> are you trying to do to trigger DownloadCommand when SSVM is not up?
>
> Thanks
> -min
>
> On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com>
> wrote:
>
> >No, that's not the problem. The problem is that the DownloadCommand
> >is supposed to go to the SSVM, but it sending it to the
> >LocalHostEndPoint which is the mgmt server. So the mgmt server can
> >not create /var/cloudstack/mnt/
> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt setup
> >so that it does not have sudo access. A standard cloudstack install
> >has sudo access so it will create those folders and mount on the mgmt
> >server, but it shouldn't.
> >
> >Please don't tell me we made a change so that the mgmt server is
> >expected to have NFS access to all secondary storages.
> >
> >Darren
> >
> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com> wrote:
> >> Please check if you can
> >> create
> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> and mount nfs point.
>
>
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Min Chen <mi...@citrix.com>.
Hi Darren,
The logic of sending command to Local in case of no SSVM is up is to
support S3 as secondary storage. In case of S3 as secondary storage,
system vm template is automatically downloaded to S3 when S3 is added into
CloudStack, unlike NFS secondary storage case where we used a script to
pre-populate system vm template. The thing I don't get is: what scenario
are you trying to do to trigger DownloadCommand when SSVM is not up?
Thanks
-min
On 10/31/13 9:32 AM, "Darren Shepherd" <da...@gmail.com> wrote:
>No, that's not the problem. The problem is that the DownloadCommand
>is supposed to go to the SSVM, but it sending it to the
>LocalHostEndPoint which is the mgmt server. So the mgmt server can
>not create /var/cloudstack/mnt/
>secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt setup
>so that it does not have sudo access. A standard cloudstack install
>has sudo access so it will create those folders and mount on the mgmt
>server, but it shouldn't.
>
>Please don't tell me we made a change so that the mgmt server is
>expected to have NFS access to all secondary storages.
>
>Darren
>
>On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com> wrote:
>> Please check if you can
>> create
>>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> and mount nfs point.
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Darren Shepherd <da...@gmail.com>.
No, that's not the problem. The problem is that the DownloadCommand
is supposed to go to the SSVM, but it sending it to the
LocalHostEndPoint which is the mgmt server. So the mgmt server can
not create /var/cloudstack/mnt/
secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011. I have my mgmt setup
so that it does not have sudo access. A standard cloudstack install
has sudo access so it will create those folders and mount on the mgmt
server, but it shouldn't.
Please don't tell me we made a change so that the mgmt server is
expected to have NFS access to all secondary storages.
Darren
On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU <us...@gmail.com> wrote:
> Please check if you can
> create /var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> and mount nfs point.
Re: Sending to LocalHostEndpoint, Is there any resolution for this?
Posted by Wei ZHOU <us...@gmail.com>.
Please check if you can
create /var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
and mount nfs point.