You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chandni Singh <ch...@datatorrent.com> on 2015/12/04 03:45:32 UTC
FileAccess API in Malhar
A FileAccess API has been added to Malhar and few implementations of the
API like TFileImpl, DTFileImpl, etc have been added as well.
I am trying to use this API in ManagedState. However I need few more
capabilities:
1. A method which tells whether a file exists on the disk.
2. A method which returns list of files in the directory.
How should I approach this?
1. Modify FileAccess API and add these methods since I think this is a
common functionality and maybe needed for other places where FileAccess is
used
2. Add another Interface, say FileAccessAndMore ( can't think of a suitable
name :) ) which has the above 2 methods. Make the FileAccessFSImpl (base
class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
ManagedState will work with FileAccessAndMore.
Please let me know your recommendations.
Thanks,
Chandni
Re: FileAccess API in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
No didn't consider failure scenarios. You are right cannot rely on the
existence of the file.
On Sun, Dec 6, 2015 at 3:21 PM, Thomas Weise <th...@datatorrent.com> wrote:
> Chandni,
>
> Is it safe to rely on the existence of the file in your use case? Are the
> failure scenarios considered?
>
> --
> sent from mobile
> On Dec 6, 2015 3:04 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
>
> > For now, I have add the 2 methods in FileAccess :
> > 1. If a file exists under a bucket dir
> > 2. list all the files in the bucket dir
> >
> > If later we need more, we can think about exposing the
> > FileSystem/FileContext.
> >
> > Created the pull request:
> > https://github.com/apache/incubator-apex-malhar/pull/119
> >
> >
> > On Fri, Dec 4, 2015 at 3:56 PM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > At this point we should probably consider the DFS as the abstraction?
> The
> > > original purpose of the wrapper was to abstract it to be able to use a
> > > different file API.
> > >
> > > --
> > > sent from mobile
> > > On Dec 4, 2015 3:53 PM, "Chandni Singh" <ch...@datatorrent.com>
> wrote:
> > >
> > > > Can I add a FileContext getFileContext(...) to the FileAccess API?
> > > >
> > > > I want to be able to access more functions that are provided by
> > > > FileContext. Either we keep on adding these methods to FileAccess or
> > > > provide access to the underlying object.
> > > >
> > > > Siyuan suggested allowing access to FileContext and not FileSystem as
> > > > FileContext can not be closed.
> > > >
> > > > Thanks,
> > > > Chandni
> > > >
> > > > On Fri, Dec 4, 2015 at 4:20 AM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > > wrote:
> > > >
> > > > > My preference would be to add these functions in FileAccess itself
> > > other
> > > > > than creating a new interface.
> > > > >
> > > > > - Tushar.
> > > > >
> > > > > On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > A FileAccess API has been added to Malhar and few implementations
> > of
> > > > the
> > > > > > API like TFileImpl, DTFileImpl, etc have been added as well.
> > > > > >
> > > > > > I am trying to use this API in ManagedState. However I need few
> > more
> > > > > > capabilities:
> > > > > > 1. A method which tells whether a file exists on the disk.
> > > > > > 2. A method which returns list of files in the directory.
> > > > > >
> > > > > > How should I approach this?
> > > > > >
> > > > > > 1. Modify FileAccess API and add these methods since I think this
> > is
> > > a
> > > > > > common functionality and maybe needed for other places where
> > > FileAccess
> > > > > is
> > > > > > used
> > > > > >
> > > > > > 2. Add another Interface, say FileAccessAndMore ( can't think of
> a
> > > > > suitable
> > > > > > name :) ) which has the above 2 methods. Make the
> FileAccessFSImpl
> > > > (base
> > > > > > class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> > > > > > ManagedState will work with FileAccessAndMore.
> > > > > >
> > > > > > Please let me know your recommendations.
> > > > > >
> > > > > > Thanks,
> > > > > > Chandni
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: FileAccess API in Malhar
Posted by Thomas Weise <th...@datatorrent.com>.
Chandni,
Is it safe to rely on the existence of the file in your use case? Are the
failure scenarios considered?
--
sent from mobile
On Dec 6, 2015 3:04 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
> For now, I have add the 2 methods in FileAccess :
> 1. If a file exists under a bucket dir
> 2. list all the files in the bucket dir
>
> If later we need more, we can think about exposing the
> FileSystem/FileContext.
>
> Created the pull request:
> https://github.com/apache/incubator-apex-malhar/pull/119
>
>
> On Fri, Dec 4, 2015 at 3:56 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > At this point we should probably consider the DFS as the abstraction? The
> > original purpose of the wrapper was to abstract it to be able to use a
> > different file API.
> >
> > --
> > sent from mobile
> > On Dec 4, 2015 3:53 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
> >
> > > Can I add a FileContext getFileContext(...) to the FileAccess API?
> > >
> > > I want to be able to access more functions that are provided by
> > > FileContext. Either we keep on adding these methods to FileAccess or
> > > provide access to the underlying object.
> > >
> > > Siyuan suggested allowing access to FileContext and not FileSystem as
> > > FileContext can not be closed.
> > >
> > > Thanks,
> > > Chandni
> > >
> > > On Fri, Dec 4, 2015 at 4:20 AM, Tushar Gosavi <tu...@datatorrent.com>
> > > wrote:
> > >
> > > > My preference would be to add these functions in FileAccess itself
> > other
> > > > than creating a new interface.
> > > >
> > > > - Tushar.
> > > >
> > > > On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <
> chandni@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > A FileAccess API has been added to Malhar and few implementations
> of
> > > the
> > > > > API like TFileImpl, DTFileImpl, etc have been added as well.
> > > > >
> > > > > I am trying to use this API in ManagedState. However I need few
> more
> > > > > capabilities:
> > > > > 1. A method which tells whether a file exists on the disk.
> > > > > 2. A method which returns list of files in the directory.
> > > > >
> > > > > How should I approach this?
> > > > >
> > > > > 1. Modify FileAccess API and add these methods since I think this
> is
> > a
> > > > > common functionality and maybe needed for other places where
> > FileAccess
> > > > is
> > > > > used
> > > > >
> > > > > 2. Add another Interface, say FileAccessAndMore ( can't think of a
> > > > suitable
> > > > > name :) ) which has the above 2 methods. Make the FileAccessFSImpl
> > > (base
> > > > > class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> > > > > ManagedState will work with FileAccessAndMore.
> > > > >
> > > > > Please let me know your recommendations.
> > > > >
> > > > > Thanks,
> > > > > Chandni
> > > > >
> > > >
> > >
> >
>
Re: FileAccess API in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
For now, I have add the 2 methods in FileAccess :
1. If a file exists under a bucket dir
2. list all the files in the bucket dir
If later we need more, we can think about exposing the
FileSystem/FileContext.
Created the pull request:
https://github.com/apache/incubator-apex-malhar/pull/119
On Fri, Dec 4, 2015 at 3:56 PM, Thomas Weise <th...@datatorrent.com> wrote:
> At this point we should probably consider the DFS as the abstraction? The
> original purpose of the wrapper was to abstract it to be able to use a
> different file API.
>
> --
> sent from mobile
> On Dec 4, 2015 3:53 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
>
> > Can I add a FileContext getFileContext(...) to the FileAccess API?
> >
> > I want to be able to access more functions that are provided by
> > FileContext. Either we keep on adding these methods to FileAccess or
> > provide access to the underlying object.
> >
> > Siyuan suggested allowing access to FileContext and not FileSystem as
> > FileContext can not be closed.
> >
> > Thanks,
> > Chandni
> >
> > On Fri, Dec 4, 2015 at 4:20 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > My preference would be to add these functions in FileAccess itself
> other
> > > than creating a new interface.
> > >
> > > - Tushar.
> > >
> > > On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <chandni@datatorrent.com
> >
> > > wrote:
> > >
> > > > A FileAccess API has been added to Malhar and few implementations of
> > the
> > > > API like TFileImpl, DTFileImpl, etc have been added as well.
> > > >
> > > > I am trying to use this API in ManagedState. However I need few more
> > > > capabilities:
> > > > 1. A method which tells whether a file exists on the disk.
> > > > 2. A method which returns list of files in the directory.
> > > >
> > > > How should I approach this?
> > > >
> > > > 1. Modify FileAccess API and add these methods since I think this is
> a
> > > > common functionality and maybe needed for other places where
> FileAccess
> > > is
> > > > used
> > > >
> > > > 2. Add another Interface, say FileAccessAndMore ( can't think of a
> > > suitable
> > > > name :) ) which has the above 2 methods. Make the FileAccessFSImpl
> > (base
> > > > class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> > > > ManagedState will work with FileAccessAndMore.
> > > >
> > > > Please let me know your recommendations.
> > > >
> > > > Thanks,
> > > > Chandni
> > > >
> > >
> >
>
Re: FileAccess API in Malhar
Posted by Thomas Weise <th...@datatorrent.com>.
At this point we should probably consider the DFS as the abstraction? The
original purpose of the wrapper was to abstract it to be able to use a
different file API.
--
sent from mobile
On Dec 4, 2015 3:53 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
> Can I add a FileContext getFileContext(...) to the FileAccess API?
>
> I want to be able to access more functions that are provided by
> FileContext. Either we keep on adding these methods to FileAccess or
> provide access to the underlying object.
>
> Siyuan suggested allowing access to FileContext and not FileSystem as
> FileContext can not be closed.
>
> Thanks,
> Chandni
>
> On Fri, Dec 4, 2015 at 4:20 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > My preference would be to add these functions in FileAccess itself other
> > than creating a new interface.
> >
> > - Tushar.
> >
> > On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <ch...@datatorrent.com>
> > wrote:
> >
> > > A FileAccess API has been added to Malhar and few implementations of
> the
> > > API like TFileImpl, DTFileImpl, etc have been added as well.
> > >
> > > I am trying to use this API in ManagedState. However I need few more
> > > capabilities:
> > > 1. A method which tells whether a file exists on the disk.
> > > 2. A method which returns list of files in the directory.
> > >
> > > How should I approach this?
> > >
> > > 1. Modify FileAccess API and add these methods since I think this is a
> > > common functionality and maybe needed for other places where FileAccess
> > is
> > > used
> > >
> > > 2. Add another Interface, say FileAccessAndMore ( can't think of a
> > suitable
> > > name :) ) which has the above 2 methods. Make the FileAccessFSImpl
> (base
> > > class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> > > ManagedState will work with FileAccessAndMore.
> > >
> > > Please let me know your recommendations.
> > >
> > > Thanks,
> > > Chandni
> > >
> >
>
Re: FileAccess API in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
Can I add a FileContext getFileContext(...) to the FileAccess API?
I want to be able to access more functions that are provided by
FileContext. Either we keep on adding these methods to FileAccess or
provide access to the underlying object.
Siyuan suggested allowing access to FileContext and not FileSystem as
FileContext can not be closed.
Thanks,
Chandni
On Fri, Dec 4, 2015 at 4:20 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> My preference would be to add these functions in FileAccess itself other
> than creating a new interface.
>
> - Tushar.
>
> On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > A FileAccess API has been added to Malhar and few implementations of the
> > API like TFileImpl, DTFileImpl, etc have been added as well.
> >
> > I am trying to use this API in ManagedState. However I need few more
> > capabilities:
> > 1. A method which tells whether a file exists on the disk.
> > 2. A method which returns list of files in the directory.
> >
> > How should I approach this?
> >
> > 1. Modify FileAccess API and add these methods since I think this is a
> > common functionality and maybe needed for other places where FileAccess
> is
> > used
> >
> > 2. Add another Interface, say FileAccessAndMore ( can't think of a
> suitable
> > name :) ) which has the above 2 methods. Make the FileAccessFSImpl (base
> > class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> > ManagedState will work with FileAccessAndMore.
> >
> > Please let me know your recommendations.
> >
> > Thanks,
> > Chandni
> >
>
Re: FileAccess API in Malhar
Posted by Tushar Gosavi <tu...@datatorrent.com>.
My preference would be to add these functions in FileAccess itself other
than creating a new interface.
- Tushar.
On Fri, Dec 4, 2015 at 8:15 AM, Chandni Singh <ch...@datatorrent.com>
wrote:
> A FileAccess API has been added to Malhar and few implementations of the
> API like TFileImpl, DTFileImpl, etc have been added as well.
>
> I am trying to use this API in ManagedState. However I need few more
> capabilities:
> 1. A method which tells whether a file exists on the disk.
> 2. A method which returns list of files in the directory.
>
> How should I approach this?
>
> 1. Modify FileAccess API and add these methods since I think this is a
> common functionality and maybe needed for other places where FileAccess is
> used
>
> 2. Add another Interface, say FileAccessAndMore ( can't think of a suitable
> name :) ) which has the above 2 methods. Make the FileAccessFSImpl (base
> class of TFileImpl & DTFileImpl) implement FileAccessAndMore.
> ManagedState will work with FileAccessAndMore.
>
> Please let me know your recommendations.
>
> Thanks,
> Chandni
>