hadoop need help please suggest

[snehal nagmote <na...@gmail.com>]

Hello Sir,

I have some doubts, please help me.
we have requirement of scalable storage system, we have developed one
agro-advisory system in which farmers will sent the crop pictures
particularly in sequential manner some
6-7 photos of 3-4 kb each would be stored in storage server and these photos
would be read sequentially by scientist to detect the problem, writing to
images would not be done.

So for storing these images we  are using hadoop file system, is it feasible
to use hadoop
file system for the same purpose.

As also the images are of only 3-4 kb and hadoop reads the data in blocks of
size 64 mb
how can we increase the performance, what could be the tricks and tweaks
that should be done to use hadoop for such kind of purpose.

Next problem is as hadoop stores all the metadata in memory,can we use some
mechanism to store the files in the block of some greater size because as
the files would be of small size,so it will store the lots metadata and will
overflow the main memory
please suggest what could be done


regards,
Snehal

Re: hadoop need help please suggest

[Snehal Nagmote <na...@gmail.com>]

Sorry for the inconvenience caused ...I will not spam core dev.
Scale we are thinking in terms of more nodes in coming future can go to
petabytes of data
Can you please give some pointers for handling the same issue, i am quite
new to hadoop

Regards,
Snehal




Raghu Angadi wrote:
> 
> 
> What is scale you are thinking of? (10s, 100s or more nodes)?
> 
> The memory for metadata at NameNode you mentioned is that main issue 
> with small files. There are multiple alternatives for the dealing with 
> that. This issue is discussed many times here.
> 
> Also please use core-user@ id alone for asking for help.. you don't need 
> to send to core-devel@
> 
> Raghu.
> 
> snehal nagmote wrote:
>> Hello Sir,
>> 
>> I have some doubts, please help me.
>> we have requirement of scalable storage system, we have developed one
>> agro-advisory system in which farmers will sent the crop pictures
>> particularly in sequential manner some
>> 6-7 photos of 3-4 kb each would be stored in storage server and these
>> photos
>> would be read sequentially by scientist to detect the problem, writing to
>> images would not be done.
>> 
>> So for storing these images we  are using hadoop file system, is it
>> feasible
>> to use hadoop
>> file system for the same purpose.
>> 
>> As also the images are of only 3-4 kb and hadoop reads the data in blocks
>> of
>> size 64 mb
>> how can we increase the performance, what could be the tricks and tweaks
>> that should be done to use hadoop for such kind of purpose.
>> 
>> Next problem is as hadoop stores all the metadata in memory,can we use
>> some
>> mechanism to store the files in the block of some greater size because as
>> the files would be of small size,so it will store the lots metadata and
>> will
>> overflow the main memory
>> please suggest what could be done
>> 
>> 
>> regards,
>> Snehal
>> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/hadoop-need-help-please-suggest-tp22666530p22721718.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: hadoop need help please suggest

[schubert zhang <zs...@gmail.com>]

small file bring tow problem: memory of namenode and the utilization ratio
of storage.

I think following choice:1. use bigtable
2. archive in application layer.

On Wed, Mar 25, 2009 at 1:37 AM, Raghu Angadi <ra...@yahoo-inc.com> wrote:

>
> What is scale you are thinking of? (10s, 100s or more nodes)?
>
> The memory for metadata at NameNode you mentioned is that main issue with
> small files. There are multiple alternatives for the dealing with that. This
> issue is discussed many times here.
>
> Also please use core-user@ id alone for asking for help.. you don't need
> to send to core-devel@
>
> Raghu.
>
>
> snehal nagmote wrote:
>
>> Hello Sir,
>>
>> I have some doubts, please help me.
>> we have requirement of scalable storage system, we have developed one
>> agro-advisory system in which farmers will sent the crop pictures
>> particularly in sequential manner some
>> 6-7 photos of 3-4 kb each would be stored in storage server and these
>> photos
>> would be read sequentially by scientist to detect the problem, writing to
>> images would not be done.
>>
>> So for storing these images we  are using hadoop file system, is it
>> feasible
>> to use hadoop
>> file system for the same purpose.
>>
>> As also the images are of only 3-4 kb and hadoop reads the data in blocks
>> of
>> size 64 mb
>> how can we increase the performance, what could be the tricks and tweaks
>> that should be done to use hadoop for such kind of purpose.
>>
>> Next problem is as hadoop stores all the metadata in memory,can we use
>> some
>> mechanism to store the files in the block of some greater size because as
>> the files would be of small size,so it will store the lots metadata and
>> will
>> overflow the main memory
>> please suggest what could be done
>>
>>
>> regards,
>> Snehal
>>
>>
>

Re: hadoop need help please suggest

[Raghu Angadi <ra...@yahoo-inc.com>]

What is scale you are thinking of? (10s, 100s or more nodes)?

The memory for metadata at NameNode you mentioned is that main issue 
with small files. There are multiple alternatives for the dealing with 
that. This issue is discussed many times here.

Also please use core-user@ id alone for asking for help.. you don't need 
to send to core-devel@

Raghu.

snehal nagmote wrote:
> Hello Sir,
> 
> I have some doubts, please help me.
> we have requirement of scalable storage system, we have developed one
> agro-advisory system in which farmers will sent the crop pictures
> particularly in sequential manner some
> 6-7 photos of 3-4 kb each would be stored in storage server and these photos
> would be read sequentially by scientist to detect the problem, writing to
> images would not be done.
> 
> So for storing these images we  are using hadoop file system, is it feasible
> to use hadoop
> file system for the same purpose.
> 
> As also the images are of only 3-4 kb and hadoop reads the data in blocks of
> size 64 mb
> how can we increase the performance, what could be the tricks and tweaks
> that should be done to use hadoop for such kind of purpose.
> 
> Next problem is as hadoop stores all the metadata in memory,can we use some
> mechanism to store the files in the block of some greater size because as
> the files would be of small size,so it will store the lots metadata and will
> overflow the main memory
> please suggest what could be done
> 
> 
> regards,
> Snehal
>