You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@airavata.apache.org by "ushaheu (via GitHub)" <gi...@apache.org> on 2023/03/20 16:14:59 UTC
[GitHub] [airavata-mft] ushaheu opened a new issue, #92: Define Transfer Location in Database Dynamically
ushaheu opened a new issue, #92:
URL: https://github.com/apache/airavata-mft/issues/92
I would like to know if there is a feature to define multiple transfer paths in database. In my use case I have over 6000 different paths I want to pick files from and move to different locations. Is there a way to dynamically define my paths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu closed issue #92: Setting Up Airavata for File Transfer - Java
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu closed issue #92: Setting Up Airavata for File Transfer - Java
URL: https://github.com/apache/airavata-mft/issues/92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478523250
Hi @DImuthuUpe thanks once again for your support.
I have written a small application for a local transfer on my laptop from one directory to another. I first of all created the storage entry for local as commented in my code. Then I used the sample you shared. However, the file was not transferred as expected.
`public class APIManager {
public static void main(String[] args) throws InterruptedException {
MFTApiClient mftApiClient = MFTApiClient.MFTApiClientBuilder.newBuilder()
.withResourceServicePort(7003)
.withSecretServicePort(7003)
.withTransferServicePort(7003).build();
//mftApiClient.getStorageServiceClient().local().createLocalStorage(LocalStorageCreateRequest.newBuilder().setStorageId(UUID.randomUUID().toString()).setName("Mac Local").build());
String s3StorageId = "b6336695-7127-4c84-9ea4-c6a157032f32";
String scpStorageId = "50292023-f4ff-496f-8382-e5c20632244e";
String localSourcePath = "/Users/ushahembaukange/Downloads/visa-clearing-2021-09-03-1731c3c2-febe-43ae-b832-f91d043fc173.ecf";
String localDestinationPath = "/Users/ushahembaukange/Documents/filetransfer/visa-clearing-2021-09-03-1731c3c2-febe-43ae-b832-f91d043fc173.ecf";
SecretForStorage scpSecret = mftApiClient.getStorageServiceClient()
.common()
.getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(scpStorageId).build());
SecretForStorage s3Secret = mftApiClient.getStorageServiceClient()
.common()
.getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(s3StorageId).build());
EndpointPaths endpoints = EndpointPaths.newBuilder()
.setSourcePath(localSourcePath)
.setDestinationPath(localDestinationPath).build();
TransferApiResponse transferResponse = mftApiClient.getTransferClient().submitTransfer(TransferApiRequest.newBuilder()
.setSourceSecretId(scpSecret.getSecretId())
.setSourceStorageId(scpStorageId).setDestinationSecretId(s3Secret.getSecretId())
.setDestinationStorageId(s3StorageId).addEndpointPaths(endpoints).build()); // You can add many endpointpaths into one request
String transferId = transferResponse.getTransferId();
System.out.println("Transfer Id: " + transferId);
// Monitoring transfer progress. You can write your own logic to do this
for (int i = 0; i < 10; i ++) {
TransferStateSummaryResponse transferStateSummary = mftApiClient.getTransferClient()
.getTransferStateSummary(
TransferStateApiRequest.newBuilder().setTransferId(transferId).build());
System.out.println("Transfer state: " + transferStateSummary.getPercentage());
Thread.sleep(1000);
}
}
}`
Kindly find the logs from `mft log`
<img width="1438" alt="Screenshot 2023-03-21 at 21 07 19" src="https://user-images.githubusercontent.com/20520399/226729876-907c6b03-b181-4314-ad99-7554ed69a647.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476901399
Thanks once again @DImuthuUpe and @smarru I have been able to run the steps 1 to 4. I attached screenshots. I will be waiting for your guidance on the client
<img width="1434" alt="Screenshot 2023-03-20 at 21 13 08" src="https://user-images.githubusercontent.com/20520399/226460666-bfab1f7f-6934-4bae-8ec7-fbe4bbda81ef.png">
<img width="825" alt="Screenshot 2023-03-20 at 21 14 32" src="https://user-images.githubusercontent.com/20520399/226460683-ccb8fd68-6177-4633-85cd-069d8d339c76.png">
<img width="1007" alt="Screenshot 2023-03-20 at 21 36 46" src="https://user-images.githubusercontent.com/20520399/226460704-900c097a-f93f-4b60-a472-d937c2af3ea8.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478239914
Great. Looks like MFT running fine now. You might have to get a pull of the MFT repository and rebuild through maven before moving into next step. After you have MFT built,
1. Run `mft storage add` to add both SCP and S3 endpoints. This is a one-time step and you can reuse the storage you have added in multiple transfers.
2. Run `mft storage list` to list the storages you have already registered. Notice that there is a storage id for each storage.
Here is the java code to submit transfers for the storage you have added. You need to replace storage ids with the ids you got
`
package org.apache.airavata.mft.api.client;
import org.apache.airavata.mft.api.service.*;
import org.apache.airavata.mft.resource.stubs.storage.common.SecretForStorage;
import org.apache.airavata.mft.resource.stubs.storage.common.SecretForStorageGetRequest;
public class Test {
public static void main(String args[]) throws InterruptedException {
String s3StorageId = "07e4865f-7b35-4c8d-a977-81c5ba0f6ff4";
String scpStorageId = "5344d7f6-1ab6-458b-8e15-47108d7b5ead";
String scpSourcePath = "/home/exouser/a.txt";
String s3DestinationPath = "a.txt";
MFTApiClient client = MFTApiClient.MFTApiClientBuilder.newBuilder()
.withResourceServicePort(7003)
.withSecretServicePort(7003)
.withTransferServicePort(7003).build();
SecretForStorage scpSecret = client.getStorageServiceClient()
.common()
.getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(scpStorageId).build());
SecretForStorage s3Secret = client.getStorageServiceClient()
.common()
.getSecretForStorage(SecretForStorageGetRequest.newBuilder().setStorageId(s3StorageId).build());
EndpointPaths endpoints = EndpointPaths.newBuilder()
.setSourcePath(scpSourcePath)
.setDestinationPath(s3DestinationPath).build();
TransferApiResponse transferResponse = client.getTransferClient().submitTransfer(TransferApiRequest.newBuilder()
.setSourceSecretId(scpSecret.getSecretId())
.setSourceStorageId(scpStorageId).setDestinationSecretId(s3Secret.getSecretId())
.setDestinationStorageId(s3StorageId).addEndpointPaths(endpoints).build()); // You can a many endpointpaths into one request
String transferId = transferResponse.getTransferId();
System.out.println("Transfer Id: " + transferId);
// Monitoring transfer progress. You can write your own logic to do this
for (int i = 0; i < 10; i ++) {
TransferStateSummaryResponse transferStateSummary = client.getTransferClient()
.getTransferStateSummary(
TransferStateApiRequest.newBuilder().setTransferId(transferId).build());
System.out.println("Transfer state: " + transferStateSummary.getState());
Thread.sleep(1000);
}
}
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1477960291
Thanks alot @DImuthuUpe for your support and kind words. I have run the commands above. Kindly find attached the logs.
<img width="1437" alt="Screenshot 2023-03-21 at 15 37 09" src="https://user-images.githubusercontent.com/20520399/226640946-bd8b1381-9702-4662-979a-a3ac5433f58a.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478528187
Can you do
`mft stop`
`mft init`
and retry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476570904
@ushaheu Do you want to move data within your local workstation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476745077
Thanks alot for your response @smarru. My application is a java application and I have mastery of Java not Python hence my preference for Java. I will be glad to get the response from @DImuthuUpe on how to integrate airavata mft with Java/Spring boot. Thanks for the great work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478534446
No it is not required. We need to investigate why the agent was down. Can you send the full log file available in ~/.mft/Standalone-Service-0.01/logs/airavata.log ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478314320
I appreciate you @DImuthuUpe. I will try it out and provide an update as soon as I get it right. Thanks alot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1477841608
@ushaheu This is great. I encountered a couple of bugs in SCP transport and updated the binary version with the fix. Can you please update mft-cli and reinstall mft? Here are the commands
`pip3.10 install --upgrade airavata-mft-cli`
`mft update`
Above commands will remove your existing mft setup and reinstall the latest version. To verify if you have it running properly, please send the output of
`mft log`
I am still working on the java snippet and I'll send you it as soon as it is available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478305026
Yes they are. Please refer to sample APIs like below.
```
s3Storage = client.getStorageServiceClient().s3().createS3Storage()
s3Secret = client.getSecretServiceClient().s3().createS3Secret()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476759610
Thank you very much for the detailed information @DImuthuUpe. I will go through these steps and come back to you for the client code. Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478532591
>
Thank you very much @DImuthuUpe . I was able to transfer now. Is there any reason we had to do that? Is it a command I will need to run often.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478532558
If you need help, I can jump into a zoom breifly. Contact me through dimuthu.upeksha2@gmail.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] smarru commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "smarru (via GitHub)" <gi...@apache.org>.
smarru commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476737345
@ushaheu in general, MFT seems to be a good fit for what you want to achieve. It can translate the protocol from SCP to S3, allowing you to plug in different credentials. I will let @DImuthuUpe respond on the java client. To explore options, will a Python client also work, or do you prefer the client to be in Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1478269182
Thanks alot for this support @DImuthuUpe I will implement this and provide update as soon as am done. I have two quick question, is there a possibility that `mft storage add` and `mft storage list` will be exposed as an api? Then can I transfer to s3 bucket from my local drive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] smarru commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "smarru (via GitHub)" <gi...@apache.org>.
smarru commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476708726
@ushaheu I see you will need to push data to S3, what will be the source protocol? A server with SCP/SSH access or other cloud storages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476713597
Thanks for your response @smarru my source protocol will be SCP. Would you be kind to also share a sample java snippet or javadoc to guide me use the mft with Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] ushaheu commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "ushaheu (via GitHub)" <gi...@apache.org>.
ushaheu commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476676862
@DImuthuUpe I want to move data from my server where I generate reports to an s3 bucket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airavata-mft] DImuthuUpe commented on issue #92: Define Transfer Location in Database Dynamically
Posted by "DImuthuUpe (via GitHub)" <gi...@apache.org>.
DImuthuUpe commented on issue #92:
URL: https://github.com/apache/airavata-mft/issues/92#issuecomment-1476757124
@ushaheu seems like you are planning to integrate MFT into your application. The obvious way is to have MFT running as a separate process and connect to it through its gRPC API. It does not need spring to be running in your application. Here are the steps you need to follow
1. Start MFT by following the steps https://github.com/apache/airavata-mft#download-and-install
2. Build MFT Project using maven on your laptop (You need to do this as you plan to programmatically access MFT)
3. Import this MFT client module into your maven project https://github.com/apache/airavata-mft/blob/master/api/client/pom.xml
4. Use the MFTClient to https://github.com/apache/airavata-mft/blob/master/api/client/src/main/java/org/apache/airavata/mft/api/client/MFTApiClient.java to talk to MFT
Update us once you managed to successfully complete the above steps. I will send a client usage sample code quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@airavata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org