You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Steve Lewis <lo...@gmail.com> on 2011/09/14 18:24:52 UTC
Has anyone ever written a file system where the data is held in resources
When writing tests it is useful to keep all data in resources since this
makes automatic execution easier.
The structure of a set of resources should make it easy to have a schema
such as res:// to look there -
Has anyone already done the work.
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Has anyone ever written a file system where the data is held in resources
Posted by Joey Echeverria <jo...@cloudera.com>.
To add to what Kevin said, you'll be writing a class that extends FileSystem.
-Joey
On Wed, Sep 14, 2011 at 1:08 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> You would probably have to implement your own Hadoop filesystem similar to
> S3 and KFS integrate.
> I looked at it a while back and it didn't seem insanely difficult …
> Kevin
>
> On Wed, Sep 14, 2011 at 9:47 AM, Steve Lewis <lo...@gmail.com> wrote:
>>
>> No - the issue is I want is I want Hadoop to read resources as input files
>> as if they were in hdfs - I know how to read resources as
>> input streams but I don't know how to get a Hadoop file system which will
>> treat a Path like res://myclass/myresource.txt as useful and
>> give me an FSInputStream (rather that a simple InputStream)
>>
>> On Wed, Sep 14, 2011 at 9:38 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>>>
>>> You can already do this with the JAR file format… if you load a resource
>>> via path it uses the class loader system to find it in all available jars.
>>> Kevin
>>>
>>> On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis <lo...@gmail.com>
>>> wrote:
>>>>
>>>> When writing tests it is useful to keep all data in resources since this
>>>> makes automatic execution easier.
>>>> The structure of a set of resources should make it easy to have a schema
>>>> such as res:// to look there -
>>>> Has anyone already done the work.
>>>>
>>>> --
>>>> Steven M. Lewis PhD
>>>> 4221 105th Ave NE
>>>> Kirkland, WA 98033
>>>> 206-384-1340 (cell)
>>>> Skype lordjoe_com
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>>
>>> Location: San Francisco, CA
>>> Skype: burtonator
>>>
>>> Skype-in: (415) 871-0687
>>
>>
>>
>> --
>> Steven M. Lewis PhD
>> 4221 105th Ave NE
>> Kirkland, WA 98033
>> 206-384-1340 (cell)
>> Skype lordjoe_com
>>
>>
>
>
>
> --
>
> Founder/CEO Spinn3r.com
>
> Location: San Francisco, CA
> Skype: burtonator
>
> Skype-in: (415) 871-0687
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: Has anyone ever written a file system where the data is held in resources
Posted by Kevin Burton <bu...@spinn3r.com>.
You would probably have to implement your own Hadoop filesystem similar to
S3 and KFS integrate.
I looked at it a while back and it didn't seem insanely difficult …
Kevin
On Wed, Sep 14, 2011 at 9:47 AM, Steve Lewis <lo...@gmail.com> wrote:
> No - the issue is I want is I want Hadoop to read resources as input files
> as if they were in hdfs - I know how to read resources as
> input streams but I don't know how to get a Hadoop file system which will
> treat a Path like res://myclass/myresource.txt as useful and
> give me an FSInputStream (rather that a simple InputStream)
>
>
> On Wed, Sep 14, 2011 at 9:38 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> You can already do this with the JAR file format… if you load a resource
>> via path it uses the class loader system to find it in all available jars.
>>
>> Kevin
>>
>>
>> On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis <lo...@gmail.com>wrote:
>>
>>> When writing tests it is useful to keep all data in resources since this
>>> makes automatic execution easier.
>>> The structure of a set of resources should make it easy to have a schema
>>> such as res:// to look there -
>>> Has anyone already done the work.
>>>
>>> --
>>> Steven M. Lewis PhD
>>> 4221 105th Ave NE
>>> Kirkland, WA 98033
>>> 206-384-1340 (cell)
>>> Skype lordjoe_com
>>>
>>>
>>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>>
>> Location: *San Francisco, CA*
>> Skype: *burtonator*
>>
>> Skype-in: *(415) 871-0687*
>>
>>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
Skype-in: *(415) 871-0687*
Re: Has anyone ever written a file system where the data is held in resources
Posted by Steve Lewis <lo...@gmail.com>.
No - the issue is I want is I want Hadoop to read resources as input files
as if they were in hdfs - I know how to read resources as
input streams but I don't know how to get a Hadoop file system which will
treat a Path like res://myclass/myresource.txt as useful and
give me an FSInputStream (rather that a simple InputStream)
On Wed, Sep 14, 2011 at 9:38 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> You can already do this with the JAR file format… if you load a resource
> via path it uses the class loader system to find it in all available jars.
>
> Kevin
>
>
> On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis <lo...@gmail.com>wrote:
>
>> When writing tests it is useful to keep all data in resources since this
>> makes automatic execution easier.
>> The structure of a set of resources should make it easy to have a schema
>> such as res:// to look there -
>> Has anyone already done the work.
>>
>> --
>> Steven M. Lewis PhD
>> 4221 105th Ave NE
>> Kirkland, WA 98033
>> 206-384-1340 (cell)
>> Skype lordjoe_com
>>
>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
>
> Location: *San Francisco, CA*
> Skype: *burtonator*
>
> Skype-in: *(415) 871-0687*
>
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Has anyone ever written a file system where the data is held in resources
Posted by Kevin Burton <bu...@spinn3r.com>.
You can already do this with the JAR file format… if you load a resource via
path it uses the class loader system to find it in all available jars.
Kevin
On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis <lo...@gmail.com> wrote:
> When writing tests it is useful to keep all data in resources since this
> makes automatic execution easier.
> The structure of a set of resources should make it easy to have a schema
> such as res:// to look there -
> Has anyone already done the work.
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
Skype-in: *(415) 871-0687*