You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@airavata.apache.org by GitBox <gi...@apache.org> on 2021/03/18 22:30:30 UTC
[GitHub] [airavata-mft] machristie opened a new issue #33: File resource download hangs when trying to download file
machristie opened a new issue #33:
URL: https://github.com/apache/airavata-mft/issues/33
<!--
This form is for bug reports and feature requests ONLY!
For general questions and troubleshooting about Airavata MFT, please ask/look for answers here:
- Airavata MFT mailing list: https://lists.apache.org/list.html?dev@airavata.apache.org
Issues specific to *airavata*, *airavata-django-portal*, *airavata-custos*, should be created in the repository they belong to (e.g. https://github.com/apache/airavata)
-->
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
I'm running download_http.py to generate a download url for a file resource. When I click on the generated URL to download the file, it just hangs.
## Steps to Reproduce
[NOTE]: # ( Include details description or commands to reproduce. )
I created a SCP file resource
```json
...
{
"type": "SCP",
"resourceId": "remote-ssh-resource",
"resourceMode": "FILE",
"resourcePath": "/tmp/test1/solution_summary.json",
"storageId" : "remote-ssh-storage"
},
```
Then I generate the download URL with a python script (modified download_http.py):
```python
import grpc
import MFTApi_pb2
import MFTApi_pb2_grpc
channel = grpc.insecure_channel('localhost:7004')
stub = MFTApi_pb2_grpc.MFTApiServiceStub(channel)
download_request = MFTApi_pb2.HttpDownloadApiRequest(
# sourceStoreId ="remote-ssh-storage",
sourceResourceId = "remote-ssh-resource",
# sourceResourceId = "remote-ssh-dir-resource",
# sourceStoreId ="remote-ssh-dir-resource",
# sourcePath= "/tmp/test1/foo",
sourceToken = "local-ssh-cred",
sourceType= "SCP",
targetAgent = "agent0",
# mftAuthorizationToken = ""
)
result = stub.submitHttpDownload(download_request)
print(result)
```
When I run the script I get:
```
$ python download_http.py
url: "http://localhost:3333/4978023d-c807-46fc-989b-8f550d18021d"
targetAgent: "agent0"
```
When I load the URL in a browser, the browser just spins and never finishes loading the file.
## Expected Behaviour
[NOTE]: # ( Tell us what you did and what you expected to happen and what you instead saw. )
Loading the URL in the browser would result in a file download.
## Your Environment
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( Mention Custos Branch or release version, runtime and compiler version )
* mft branch or release version used: develop
* Operating system and version: macOS Big Sur
## Additional Context
[TIP]: # ( full error message, exception listing, stack trace, logs or other information which can assist in diagnosing the bug. )
By using logging statements, I know that it gets to trying to trying to create a session in SCPReceiver.java
https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPReceiver.java#L103
Apparently it is stuck on this line in SCPTransportUtil:
https://github.com/apache/airavata-mft/blob/develop/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPTransportUtil.java#L37
because I can log the line before but none of my logging after this line prints.
Also, I know that the SCP configuration is good since I'm able to fetch metadata for the resource:
```python
request = MFTApi_pb2.FetchResourceMetadataRequest(
resourceId= "remote-ssh-resource",
resourceType = "SCP",
resourceToken = "local-ssh-cred",
targetAgentId = "agent0",
)
response = stub.getFileResourceMetadata(request)
print(response)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] machristie commented on issue #33: File resource download hangs when trying to download file
Posted by GitBox <gi...@apache.org>.
machristie commented on issue #33:
URL: https://github.com/apache/airavata-mft/issues/33#issuecomment-812536503
I've done some debugging and I've found that for some reason the private key is invalid. But also, when the exception is thrown it is causing the thread pool mechanism to just hang, but I'm not sure why. I'll continue investigating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] machristie commented on issue #33: File resource download hangs when trying to download file
Posted by GitBox <gi...@apache.org>.
machristie commented on issue #33:
URL: https://github.com/apache/airavata-mft/issues/33#issuecomment-813556118
@DImuthuUpe I have this fixed for now but I think there are still a few issues here:
- exception handling needs to be fixed in HttpServerHandler. The [pullStatusFuture](https://github.com/apache/airavata-mft/blob/machristie/issue33/agent/src/main/java/org/apache/airavata/mft/agent/http/HttpServerHandler.java#L128) results in an Exception when the key file is not a supported format and then the connection just hangs.
- SCPReceiver/SCPSender use JSch but the rest of the codebase uses ssh4j. Would it be possible to switch SCPReceiver/SCPSender to ssh4j too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] machristie commented on issue #33: File resource download hangs when trying to download file
Posted by GitBox <gi...@apache.org>.
machristie commented on issue #33:
URL: https://github.com/apache/airavata-mft/issues/33#issuecomment-813467954
It turns out it was a private key format issue. Downloading a file uses the SCPReceiver/SCPSender classes and these classes use the JSch library instead of the ssh4j library. JSch doesn't support the newer format of OpenSSH private keys, see https://stackoverflow.com/questions/53134212/invalid-privatekey-when-using-jsch.
I was able to convert my private key to the older format by using this:
```
ssh-keygen -p -f local-ssh-cred -m pem
```
The `-m pem` argument tells openssh to write private key in the classic PEM format.
This private key is now useable with ssh4j and JSch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org