You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airavata.apache.org by Eroma Abeysinghe <er...@gmail.com> on 2016/08/01 13:08:54 UTC
Re: adding applications to Airavata
Hi Jarett,
Order is Application module --> Interface --> Deployment
1.Application module is the first step of introducing the application.
2. Interface is where we define the inputs and outputs. We can define what
type of input, whether we need it in command-line or not. data staging,
etc.... and same for the outputs of the application as well. Here we can
also define whether the application ay need additional optional inputs as
well as we need to archive all the input and output files in the
working directory.
This will be the inputs asked at experiment creation for launching a
certain application job in compute resource.
3. Deployment is used to define the location of the application to Airavata
and also the commands required to execute. Here we can also give any post
and pre commands required fort the application execution as well.
Hope above gives you an idea of the application catalog.
Please try below links as well for more information
http://airavata.readthedocs.io/en/latest/Gateway-Configurations/#AppCatalog
https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+05+-+PHP+Reference+Gateway+for+Airavata+-+Gateway+Admin+Guide#Tutorial05-PHPReferenceGatewayforAiravata-GatewayAdminGuide-TutorialV-Create&SearchApplicationModule,ApplicationInterface,andApplicationDeployment
Thanks,
Eroma
On Fri, Jul 29, 2016 at 5:05 PM, Jarett DeAngelis <ja...@bioteam.net>
wrote:
> Hi folks,
>
> I’m trying to locate documentation for the canonical way to add
> applications to Airavata. Looking at the gateway in the admin console, it
> looks like there are several steps involved:
>
> Adding an Application Module
> Adding an Application Interface
> Adding an Application Deployment
>
> In what order should these be done, and how?
>
> I’m guessing this goes something like,
>
>
> - Install application on the Airavata node, via packages or into
> /usr/local from source
> - Point Airavata to the location of the application (this is an
> “application module?”)
> - Create a set of parameters etc. for the application to use
> (“application interface?)
> - Publish the application with its set of parameters to the gateway
> (is that what an application deployment is?)
>
>
> There seems to be no way to add applications in the “modules” section that
> I can find. If you do “Create a New Application Module” you get options for
> a name, a version and description, without any mention of e.g. a binary to
> launch. Help?
>
> Thanks!
>
> Jarett DeAngelis
>
--
Thank You,
Best Regards,
Eroma
Re: File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
So, we’ve talked about this since and I am taking a crack at including the SFTP feature into PGA. This seems like an ideal framework to include: https://github.com/thephpleague/flysystem-sftp <https://github.com/thephpleague/flysystem-sftp>
I’m looking for the part of the application where the input and output types are defined — I’m guessing we would do this as another “type” like “URI” or “STRING.” Can someone give me a good place to start?
Thanks,
J
> On Sep 1, 2016, at 4:32 PM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Jarett,
>
> We have some interest from who were looking for task to contribute to Airavata. We can probably motivate interest on this task. Can you please write a detailed use cases clearly articulating the user sequence of actions you would like to see? We can then over lay that with Airavata architecture, break it down into small sprints and work through it.
>
> Thanks,
> Suresh
>
>> On Sep 1, 2016, at 4:06 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>
>> Hi Suresh,
>>
>> Just wanted to follow up on this: I see what you mean re: the file picker connected to a remote node. So, given that what we want to do is allow users to both access shared data (like databases, for example, for applications like blastn) and read and write data to their home folders, what do you think is the best course of action?
>>
>> I’m also curious what existing use cases you’re seeing that make the gateway useful currently, *without* access to permission-controlled user directories. How are people using it now?
>>
>> Ultimately, fixes for this situation that would help us would include:
>> 1) A file picker for directories *local* to the PGA gateway (because we could put shared resources like databases in a directory like this)
>> 2) The ability to SCP/SFTP, from the gateway, in and out of user-permission space, using the credentials entered into the gateway to log in.
>>
>> So, some questions:
>> Do you see either of these as being implementable in the short term?
>> Are these on your radar for the Airavata team to implement themselves?
>>
>> Thanks!
>>
>> Jarett
>>
>>
>>> On Aug 12, 2016, at 4:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>
>>>
>>>> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>
>>>> Hey Suresh,
>>>>
>>>> Thanks for starting this conversation!
>>>>
>>>> What do you think the users’ “extra one time configuration” would look like?
>>>>
>>>> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>>>>
>>>> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>>>
>>> Hi Jaratt,
>>>
>>> After thinking through this scenario, I think users providing a path to a permissible users space is a low hanging fruit. Doing a file picker (which implies reading remote files) will be an involving task since PGA does not talk to remote server directly but has to broker the file listings through Airavata. Make sense?
>>>
>>> Suresh
>>>
>>>>
>>>> Thanks!
>>>> Jarett
>>>>
>>>>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Hi Jarett,
>>>>>
>>>>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>>>>
>>>>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>>>>
>>>>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>>>>
>>>>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>>>>
>>>>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>>>>
>>>>> Cheers,
>>>>> Suresh
>>>>>
>>>>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>>>>
>>>>>
>>>>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Jarett
>>>>>
>>>>
>>>
>>
>
Re: File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
Absolutely.
Use case #1 (probably the most useful change that would be immediately feasible):
User logs into gateway using their directory service credentials; WSO2 authenticates them to FreeIPA.
User starts an experiment, gateway uses SFTP/SCP to get a directory listing from an SSH server and standard directory path (this would be defined in the “input” part of the experiment definition), a URI such as sftp://fileserver.domain.tld/home/$username
User can choose a file from this directory listing as an input parameter.
User then chooses a directory from sftp://fileserver.domain.tld/home/$username again as a place to put output files.
User kicks off experiment job, gateway uses provided input and output paths and notifies user when job is complete as usual. No results are returned through the browser in this case; instead they are written to the SFTP path.
Use case #2:
User logs into gateway as usual.
User starts an experiment, one of the “input” parameters is another directory listing, but this time of a directory local to the gateway machine, to which the Airavata service account has access.
User can choose a file from this directory listing as an input parameter.
User kicks off experiment job with other inputs and outputs as usual, gateway reads one of the input parameters from its own filesystem, returns results to browser as usual.
Ideally, we’d be able to do both of these. It would enable a number of applications that read large amounts of data in single files.
Thanks!
> On Sep 1, 2016, at 4:32 PM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Jarett,
>
> We have some interest from who were looking for task to contribute to Airavata. We can probably motivate interest on this task. Can you please write a detailed use cases clearly articulating the user sequence of actions you would like to see? We can then over lay that with Airavata architecture, break it down into small sprints and work through it.
>
> Thanks,
> Suresh
>
>> On Sep 1, 2016, at 4:06 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>
>> Hi Suresh,
>>
>> Just wanted to follow up on this: I see what you mean re: the file picker connected to a remote node. So, given that what we want to do is allow users to both access shared data (like databases, for example, for applications like blastn) and read and write data to their home folders, what do you think is the best course of action?
>>
>> I’m also curious what existing use cases you’re seeing that make the gateway useful currently, *without* access to permission-controlled user directories. How are people using it now?
>>
>> Ultimately, fixes for this situation that would help us would include:
>> 1) A file picker for directories *local* to the PGA gateway (because we could put shared resources like databases in a directory like this)
>> 2) The ability to SCP/SFTP, from the gateway, in and out of user-permission space, using the credentials entered into the gateway to log in.
>>
>> So, some questions:
>> Do you see either of these as being implementable in the short term?
>> Are these on your radar for the Airavata team to implement themselves?
>>
>> Thanks!
>>
>> Jarett
>>
>>
>>> On Aug 12, 2016, at 4:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>
>>>
>>>> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>
>>>> Hey Suresh,
>>>>
>>>> Thanks for starting this conversation!
>>>>
>>>> What do you think the users’ “extra one time configuration” would look like?
>>>>
>>>> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>>>>
>>>> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>>>
>>> Hi Jaratt,
>>>
>>> After thinking through this scenario, I think users providing a path to a permissible users space is a low hanging fruit. Doing a file picker (which implies reading remote files) will be an involving task since PGA does not talk to remote server directly but has to broker the file listings through Airavata. Make sense?
>>>
>>> Suresh
>>>
>>>>
>>>> Thanks!
>>>> Jarett
>>>>
>>>>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Hi Jarett,
>>>>>
>>>>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>>>>
>>>>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>>>>
>>>>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>>>>
>>>>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>>>>
>>>>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>>>>
>>>>> Cheers,
>>>>> Suresh
>>>>>
>>>>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>>>>
>>>>>
>>>>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Jarett
>>>>>
>>>>
>>>
>>
>
Re: File system access for user directories, etc.
Posted by Suresh Marru <sm...@apache.org>.
Hi Jarett,
We have some interest from who were looking for task to contribute to Airavata. We can probably motivate interest on this task. Can you please write a detailed use cases clearly articulating the user sequence of actions you would like to see? We can then over lay that with Airavata architecture, break it down into small sprints and work through it.
Thanks,
Suresh
> On Sep 1, 2016, at 4:06 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> Hi Suresh,
>
> Just wanted to follow up on this: I see what you mean re: the file picker connected to a remote node. So, given that what we want to do is allow users to both access shared data (like databases, for example, for applications like blastn) and read and write data to their home folders, what do you think is the best course of action?
>
> I’m also curious what existing use cases you’re seeing that make the gateway useful currently, *without* access to permission-controlled user directories. How are people using it now?
>
> Ultimately, fixes for this situation that would help us would include:
> 1) A file picker for directories *local* to the PGA gateway (because we could put shared resources like databases in a directory like this)
> 2) The ability to SCP/SFTP, from the gateway, in and out of user-permission space, using the credentials entered into the gateway to log in.
>
> So, some questions:
> Do you see either of these as being implementable in the short term?
> Are these on your radar for the Airavata team to implement themselves?
>
> Thanks!
>
> Jarett
>
>
>> On Aug 12, 2016, at 4:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>
>>
>>> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>
>>> Hey Suresh,
>>>
>>> Thanks for starting this conversation!
>>>
>>> What do you think the users’ “extra one time configuration” would look like?
>>>
>>> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>>>
>>> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>>
>> Hi Jaratt,
>>
>> After thinking through this scenario, I think users providing a path to a permissible users space is a low hanging fruit. Doing a file picker (which implies reading remote files) will be an involving task since PGA does not talk to remote server directly but has to broker the file listings through Airavata. Make sense?
>>
>> Suresh
>>
>>>
>>> Thanks!
>>> Jarett
>>>
>>>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Hi Jarett,
>>>>
>>>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>>>
>>>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>>>
>>>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>>>
>>>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>>>
>>>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>>>
>>>> Cheers,
>>>> Suresh
>>>>
>>>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>>>
>>>>
>>>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>>>
>>>>> Thanks in advance,
>>>>> Jarett
>>>>
>>>
>>
>
Re: File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
Hi Suresh,
Just wanted to follow up on this: I see what you mean re: the file picker connected to a remote node. So, given that what we want to do is allow users to both access shared data (like databases, for example, for applications like blastn) and read and write data to their home folders, what do you think is the best course of action?
I’m also curious what existing use cases you’re seeing that make the gateway useful currently, *without* access to permission-controlled user directories. How are people using it now?
Ultimately, fixes for this situation that would help us would include:
1) A file picker for directories *local* to the PGA gateway (because we could put shared resources like databases in a directory like this)
2) The ability to SCP/SFTP, from the gateway, in and out of user-permission space, using the credentials entered into the gateway to log in.
So, some questions:
Do you see either of these as being implementable in the short term?
Are these on your radar for the Airavata team to implement themselves?
Thanks!
Jarett
> On Aug 12, 2016, at 4:17 PM, Suresh Marru <sm...@apache.org> wrote:
>
>
>> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>
>> Hey Suresh,
>>
>> Thanks for starting this conversation!
>>
>> What do you think the users’ “extra one time configuration” would look like?
>>
>> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>>
>> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>
> Hi Jaratt,
>
> After thinking through this scenario, I think users providing a path to a permissible users space is a low hanging fruit. Doing a file picker (which implies reading remote files) will be an involving task since PGA does not talk to remote server directly but has to broker the file listings through Airavata. Make sense?
>
> Suresh
>
>>
>> Thanks!
>> Jarett
>>
>>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>
>>> Hi Jarett,
>>>
>>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>>
>>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>>
>>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>>
>>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>>
>>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>>
>>> Cheers,
>>> Suresh
>>>
>>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>>
>>>
>>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>>
>>>> Thanks in advance,
>>>> Jarett
>>>
>>
>
Re: File system access for user directories, etc.
Posted by Suresh Marru <sm...@apache.org>.
> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> Hey Suresh,
>
> Thanks for starting this conversation!
>
> What do you think the users’ “extra one time configuration” would look like?
>
> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>
> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
Hi Jaratt,
After thinking through this scenario, I think users providing a path to a permissible users space is a low hanging fruit. Doing a file picker (which implies reading remote files) will be an involving task since PGA does not talk to remote server directly but has to broker the file listings through Airavata. Make sense?
Suresh
>
> Thanks!
> Jarett
>
>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>
>> Hi Jarett,
>>
>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>
>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>
>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>
>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>
>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>
>> Cheers,
>> Suresh
>>
>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>
>>
>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>
>>> Hi all,
>>>
>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>
>>> Thanks in advance,
>>> Jarett
>>
>
Re: File system access for user directories, etc.
Posted by Suresh Marru <sm...@apache.org>.
> On Aug 12, 2016, at 3:32 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> And to expand on this a little — what do you think the development effort would look like for the job submission on the compute nodes to run as the users rather than as Airavata’s service account, if we were to take it farther?
The development task is small but operational implications are huge. To elaborate, gateway administrators manage the service accounts and they can debug when jobs fail. Most of these system failures are with application environments (having right dependencies and so forth). Once the jobs are submitted from user accounts, these lead a debugging nightmares exchanging lot of emails reproducing and fixing. Essentially forcing users getting familiarize to use HPC. For most of the current users who are transitioning to using gateways this should not be an issue. But as new users are encouraged to use HPC without necessarily understanding the nuts and bolts, some one else has to absorb the details.
Suresh
>
> Thanks again,
> Jarett
>
>> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>
>> Hey Suresh,
>>
>> Thanks for starting this conversation!
>>
>> What do you think the users’ “extra one time configuration” would look like?
>>
>> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>>
>> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>>
>> Thanks!
>> Jarett
>>
>>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>
>>> Hi Jarett,
>>>
>>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>>
>>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>>
>>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>>
>>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>>
>>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>>
>>> Cheers,
>>> Suresh
>>>
>>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>>
>>>
>>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>>
>>>> Thanks in advance,
>>>> Jarett
>>>
>>
>
Re: File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
And to expand on this a little — what do you think the development effort would look like for the job submission on the compute nodes to run as the users rather than as Airavata’s service account, if we were to take it farther?
Thanks again,
Jarett
> On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> Hey Suresh,
>
> Thanks for starting this conversation!
>
> What do you think the users’ “extra one time configuration” would look like?
>
> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>
> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
>
> Thanks!
> Jarett
>
>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>
>> Hi Jarett,
>>
>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>
>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>
>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>
>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>
>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>
>> Cheers,
>> Suresh
>>
>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>
>>
>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>
>>> Hi all,
>>>
>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>
>>> Thanks in advance,
>>> Jarett
>>
>
Re: File system access for user directories, etc.
Posted by Suresh Marru <sm...@apache.org>.
Hi Jarett,
On Aug 12, 2016, at 3:23 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> Hey Suresh,
>
> Thanks for starting this conversation!
>
> What do you think the users’ “extra one time configuration” would look like?
Currently gateway administrators do this task by going to credential store and generate a token and define a storage preference. A preference includes associating a credential token to a authenticated file system and specifying a location where Airavata will pull and push data into. Second when launching an input, we need to make Airavata honor this “users” space over a ‘community” space.
>
> One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
>
> Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
These are interesting ideas, I will think more and respond on this.
Suresh
>
> Thanks!
> Jarett
>
>> On Aug 8, 2016, at 2:17 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>
>> Hi Jarett,
>>
>> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>>
>> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>>
>> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>>
>> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>>
>> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>>
>> Cheers,
>> Suresh
>>
>> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>>
>>
>>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>>
>>> Hi all,
>>>
>>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>>
>>> Thanks in advance,
>>> Jarett
>>
>
Re: File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
Hey Suresh,
Thanks for starting this conversation!
What do you think the users’ “extra one time configuration” would look like?
One of the things I was thinking about was how the gateway can do things like pass a path to the compute node in order to find files. Is it possible for the gateway to produce a “file picker” sort of dialog based on what it can see in a path you give it? It runs jobs as its service account when Airavata interacts with it. You might not have to have it run *as* the user in order to produce a directory listing for acquiring and depositing files, if the user changes the permissions on a certain set of directories such that the service account can read/write to them.
Does this make sense? Would it be a large development effort to get this additional bit of UI functionality? Obviously part of the purpose of a science gateway is to avoid having to dig around on the command line sorting out what files are where.
Thanks!
Jarett
> On Aug 8, 2016, at 2:17 PM, Suresh Marru <sm...@apache.org> wrote:
>
> Hi Jarett,
>
> To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
>
> There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
>
> If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
>
> If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
>
> Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
>
> Cheers,
> Suresh
>
> [1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
>
>
>> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <jarett@bioteam.net <ma...@bioteam.net>> wrote:
>>
>> Hi all,
>>
>> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory <scp://user:password@host:~/path/to/directory> in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>>
>> Thanks in advance,
>> Jarett
>
Re: File system access for user directories, etc.
Posted by Suresh Marru <sm...@apache.org>.
Hi Jarett,
To give some context, let me start refreshing the security(authentication and authorization aspects) within Airavata.
There are clearly two distinct layers, users securely interacting with Airavata and Airavata able to security interact with storage and compute resources. The former interactions are facilitated with WSO2 IS and later managed by Airavata Credential Store. For your use case we need to brainstorm on how we bridge the two. You can read more about Credential Store here [1].
If I am understanding your scenario, users authenticate on PGA with the credentials in LDAP/FreeIPA (brokered through WSO2 IS). Airavata then allows them to use compute resources managed by roles in the IS (in the future we might replace this with more fine grained group management). All authenticated and authorized users can interact with storage and compute resources which the gateway administers have enabled them by retrieving the credentials stored in the credential store. Currently only administrators generate and configure these credentials. We have considered power users (with appropriate role privileges also using credential store directly).
If your use case is for every user, then we need to essentially have them delegate access to their personal storage space. Access protocol is not not too much of an issue to support, how seamlessly and security we can have users delegate their storage credentials is a challenge. Its not a major development effort, but its more of a usability issue. If we need to do it in a good secure way, users should be prepared to do extra one time configuration. If we need to have users do nothing, then that will lead to less secure options.
Can we outline from a usability standout what will be a acceptable credential delegation (access to their storage space) and build from there?
Cheers,
Suresh
[1] - https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf <https://scholarworks.iu.edu/dspace/bitstream/handle/2022/17379/ccgrid_2014_credential_store.pdf>
> On Aug 2, 2016, at 5:08 PM, Jarett DeAngelis <ja...@bioteam.net> wrote:
>
> Hi all,
>
> When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
>
> Thanks in advance,
> Jarett
File system access for user directories, etc.
Posted by Jarett DeAngelis <ja...@bioteam.net>.
Hi all,
When the PGA gateway is dealing with something like getting files to and from user space (like an NFS share, for example), what is the canonical way of doing this? Since the gateway is running as the Apache service account, it does not have permissions on the user's directories. Is there something standard you can put into an application interface to let it get files back and forth? Like, some kind of spot for ssh/scp URIs, for example? I dunno, putting scp://user:password@host:~/path/to/directory in for a path? Or some more nicely-designed file picker-type interface that uses the PGA user’s logged in credentials (in our case, from LDAP/FreeIPA)?
Thanks in advance,
Jarett