You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Ralph Soika <ra...@imixs.com> on 2017/09/04 17:03:22 UTC
Is Hadoop basically not suitable for a photo archive?
Hi,
I know that the issue around the small-file problem was asked
frequently, not only in this mailing list.
I also have read already some books about Haddoop and I also started to
work with Hadoop. But still I did not really understand if Hadoop is the
right choice for my goals.
To simplify my problem domain I would like to use the use case of a
photo archive:
- An external application produces about 10 million photos in one year.
The files contain important business critical data.
- A single photo file has a size between 1 and 10 MB.
- The photos need to be stored over several years (10-30 years).
- The data store should support replication over several servers.
- A checksum-concept is needed to guarantee the data integrity of all
files over a long period of time.
- To write and read the files a Rest API is preferred.
So far Hadoop seems to be absolutely the perfect solution. But my last
requirement seems to throw Hadoop out of the race.
- The photos need to be readable with very short latency from an
external enterprise application
With Hadoop HDFS and the Web Proxy everything seems perfect. But it
seems that most of the Hadoop experts advise against this usage if the
size of my data files (1-10 MB) are well below the Hadoop block size of
64 or 128 MB.
I think I understood the concepts of HAR or sequential files.
But if I pack, for example, my files together in a large file of many
Gigabytes it is impossible to access one single photo from the Hadoop
repository in a reasonable time. It makes no sense in my eyes to pack
thousands of files into a large file just so that Hadoop jobs can handle
it better. To simply access a single file from a web interface - as in
my case - it seems to be all counterproductive.
So my question is: Is Hadoop only feasible to archive large Web-server
log files and not designed to handle big archives of small files with
also business critical data?
Thanks for your advice in advance.
Ralph
--
Re: Re: Is Hadoop basically not suitable for a photo archive?
Posted by daemeon reiydelle <da...@gmail.com>.
Kai, this is great. It is well down the path to solving the
small/object-as-file problem. Good show!
*Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872*
On Mon, Sep 4, 2017 at 8:56 PM, Zheng, Kai <ka...@intel.com> wrote:
> A nice discussion about support of small files in Hadoop.
>
>
>
> Not sure if this really helps, but I’d like to mention in Intel we
> actually has spent some time on this interesting problem domain before and
> again recently. We planned to develop a small files compaction optimization
> in the Smart Storage Management project (derived from
> https://issues.apache.org/jira/browse/HDFS-7343) that can support
> writing-a-small-file, reading-a-small-file, reading-batch-of-small-files,
> and compacting-small-files-together-in-background. These supports are
> transparent to applications but users need to use an HDFS compatible
> client. If you’re interested, please ref. the following links. We have
> rough design and plans, one important target is to support Deep Learning
> use cases that want to train lots of small samples stored into HDFS as
> files. We will implement it but your feedback would be very welcome.
>
>
>
> https://github.com/Intel-bigdata/SSM
>
> https://github.com/Intel-bigdata/SSM/blob/trunk/docs/
> small-file-solution.md
>
>
>
> Regards,
>
> Kai
>
>
>
> *From:* Hayati Gonultas [mailto:hayati.gonultas@gmail.com]
> *Sent:* Tuesday, September 05, 2017 6:06 AM
> *To:* Alexey Eremihin <a....@corp.badoo.com.invalid>; Uwe Geercken <
> uwe.geercken@web.de>
> *Cc:* Ralph Soika <ra...@imixs.com>; user@hadoop.apache.org
> *Subject:* Re: Re: Is Hadoop basically not suitable for a photo archive?
>
>
>
> I would recommend an object store such as openstack swift as another
> option.
>
>
>
> On Mon, Sep 4, 2017 at 1:09 PM Uwe Geercken <uw...@web.de> wrote:
>
> just my two cents:
>
>
>
> Maybe you can use hadoop for storing and to pack multiple files to use
> hdfs in a smarter way and at the same time store a limited amount of
> data/photos - based on time - in parallel in a different solution. I assume
> you won't need high performant access to the whole time span.
>
>
>
> Yes it would be a duplication, but maybe - without knowing all the details
> - that would be acceptable and and easy way to go for.
>
>
>
> Cheers,
>
>
>
> Uwe
>
>
>
> *Gesendet:* Montag, 04. September 2017 um 21:32 Uhr
> *Von:* "Alexey Eremihin" <a....@corp.badoo.com.INVALID>
> *An:* "Ralph Soika" <ra...@imixs.com>
> *Cc:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Betreff:* Re: Is Hadoop basically not suitable for a photo archive?
>
> Hi Ralph,
>
> In general Hadoop is able to store such data. And even Har archives can be
> used with conjunction with WebHDFS (by passing offset and limit
> attributes). What are your reading requirements? FS meta data are not
> distributed and reading the data is limited by the HDFS NameNode server
> performance. So if you would like to download files with high RPS that
> would not work well.
>
> On Monday, September 4, 2017, Ralph Soika <ra...@imixs.com> wrote:
>
> Hi,
>
> I know that the issue around the small-file problem was asked frequently,
> not only in this mailing list.
> I also have read already some books about Haddoop and I also started to
> work with Hadoop. But still I did not really understand if Hadoop is the
> right choice for my goals.
>
> To simplify my problem domain I would like to use the use case of a photo
> archive:
>
> - An external application produces about 10 million photos in one year.
> The files contain important business critical data.
> - A single photo file has a size between 1 and 10 MB.
> - The photos need to be stored over several years (10-30 years).
> - The data store should support replication over several servers.
> - A checksum-concept is needed to guarantee the data integrity of all
> files over a long period of time.
> - To write and read the files a Rest API is preferred.
>
> So far Hadoop seems to be absolutely the perfect solution. But my last
> requirement seems to throw Hadoop out of the race.
>
> - The photos need to be readable with very short latency from an external
> enterprise application
>
> With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems
> that most of the Hadoop experts advise against this usage if the size of my
> data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
>
> I think I understood the concepts of HAR or sequential files.
> But if I pack, for example, my files together in a large file of many
> Gigabytes it is impossible to access one single photo from the Hadoop
> repository in a reasonable time. It makes no sense in my eyes to pack
> thousands of files into a large file just so that Hadoop jobs can handle it
> better. To simply access a single file from a web interface - as in my case
> - it seems to be all counterproductive.
>
> So my question is: Is Hadoop only feasible to archive large Web-server log
> files and not designed to handle big archives of small files with also
> business critical data?
>
>
> Thanks for your advice in advance.
>
> Ralph
>
> --
>
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org For additional
> commands, e-mail: user-help@hadoop.apache.org
>
> --
>
> Hayati Gonultas
>
RE: Re: Is Hadoop basically not suitable for a photo archive?
Posted by "Zheng, Kai" <ka...@intel.com>.
A nice discussion about support of small files in Hadoop.
Not sure if this really helps, but I’d like to mention in Intel we actually has spent some time on this interesting problem domain before and again recently. We planned to develop a small files compaction optimization in the Smart Storage Management project (derived from https://issues.apache.org/jira/browse/HDFS-7343) that can support writing-a-small-file, reading-a-small-file, reading-batch-of-small-files, and compacting-small-files-together-in-background. These supports are transparent to applications but users need to use an HDFS compatible client. If you’re interested, please ref. the following links. We have rough design and plans, one important target is to support Deep Learning use cases that want to train lots of small samples stored into HDFS as files. We will implement it but your feedback would be very welcome.
https://github.com/Intel-bigdata/SSM
https://github.com/Intel-bigdata/SSM/blob/trunk/docs/small-file-solution.md
Regards,
Kai
From: Hayati Gonultas [mailto:hayati.gonultas@gmail.com]
Sent: Tuesday, September 05, 2017 6:06 AM
To: Alexey Eremihin <a....@corp.badoo.com.invalid>; Uwe Geercken <uw...@web.de>
Cc: Ralph Soika <ra...@imixs.com>; user@hadoop.apache.org
Subject: Re: Re: Is Hadoop basically not suitable for a photo archive?
I would recommend an object store such as openstack swift as another option.
On Mon, Sep 4, 2017 at 1:09 PM Uwe Geercken <uw...@web.de>> wrote:
just my two cents:
Maybe you can use hadoop for storing and to pack multiple files to use hdfs in a smarter way and at the same time store a limited amount of data/photos - based on time - in parallel in a different solution. I assume you won't need high performant access to the whole time span.
Yes it would be a duplication, but maybe - without knowing all the details - that would be acceptable and and easy way to go for.
Cheers,
Uwe
Gesendet: Montag, 04. September 2017 um 21:32 Uhr
Von: "Alexey Eremihin" <a....@corp.badoo.com.INVALID>>
An: "Ralph Soika" <ra...@imixs.com>>
Cc: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Betreff: Re: Is Hadoop basically not suitable for a photo archive?
Hi Ralph,
In general Hadoop is able to store such data. And even Har archives can be used with conjunction with WebHDFS (by passing offset and limit attributes). What are your reading requirements? FS meta data are not distributed and reading the data is limited by the HDFS NameNode server performance. So if you would like to download files with high RPS that would not work well.
On Monday, September 4, 2017, Ralph Soika <ra...@imixs.com>> wrote:
Hi,
I know that the issue around the small-file problem was asked frequently, not only in this mailing list.
I also have read already some books about Haddoop and I also started to work with Hadoop. But still I did not really understand if Hadoop is the right choice for my goals.
To simplify my problem domain I would like to use the use case of a photo archive:
- An external application produces about 10 million photos in one year. The files contain important business critical data.
- A single photo file has a size between 1 and 10 MB.
- The photos need to be stored over several years (10-30 years).
- The data store should support replication over several servers.
- A checksum-concept is needed to guarantee the data integrity of all files over a long period of time.
- To write and read the files a Rest API is preferred.
So far Hadoop seems to be absolutely the perfect solution. But my last requirement seems to throw Hadoop out of the race.
- The photos need to be readable with very short latency from an external enterprise application
With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems that most of the Hadoop experts advise against this usage if the size of my data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
I think I understood the concepts of HAR or sequential files.
But if I pack, for example, my files together in a large file of many Gigabytes it is impossible to access one single photo from the Hadoop repository in a reasonable time. It makes no sense in my eyes to pack thousands of files into a large file just so that Hadoop jobs can handle it better. To simply access a single file from a web interface - as in my case - it seems to be all counterproductive.
So my question is: Is Hadoop only feasible to archive large Web-server log files and not designed to handle big archives of small files with also business critical data?
Thanks for your advice in advance.
Ralph
--
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org> For additional commands, e-mail: user-help@hadoop.apache.org<ma...@hadoop.apache.org>
--
Hayati Gonultas
Re: Re: Is Hadoop basically not suitable for a photo archive?
Posted by Hayati Gonultas <ha...@gmail.com>.
I would recommend an object store such as openstack swift as another option.
On Mon, Sep 4, 2017 at 1:09 PM Uwe Geercken <uw...@web.de> wrote:
> just my two cents:
>
> Maybe you can use hadoop for storing and to pack multiple files to use
> hdfs in a smarter way and at the same time store a limited amount of
> data/photos - based on time - in parallel in a different solution. I assume
> you won't need high performant access to the whole time span.
>
> Yes it would be a duplication, but maybe - without knowing all the details
> - that would be acceptable and and easy way to go for.
>
> Cheers,
>
> Uwe
>
> *Gesendet:* Montag, 04. September 2017 um 21:32 Uhr
> *Von:* "Alexey Eremihin" <a....@corp.badoo.com.INVALID>
> *An:* "Ralph Soika" <ra...@imixs.com>
> *Cc:* "user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Betreff:* Re: Is Hadoop basically not suitable for a photo archive?
> Hi Ralph,
> In general Hadoop is able to store such data. And even Har archives can be
> used with conjunction with WebHDFS (by passing offset and limit
> attributes). What are your reading requirements? FS meta data are not
> distributed and reading the data is limited by the HDFS NameNode server
> performance. So if you would like to download files with high RPS that
> would not work well.
>
> On Monday, September 4, 2017, Ralph Soika <ra...@imixs.com> wrote:
>>
>> Hi,
>>
>> I know that the issue around the small-file problem was asked frequently,
>> not only in this mailing list.
>> I also have read already some books about Haddoop and I also started to
>> work with Hadoop. But still I did not really understand if Hadoop is the
>> right choice for my goals.
>>
>> To simplify my problem domain I would like to use the use case of a photo
>> archive:
>>
>> - An external application produces about 10 million photos in one year.
>> The files contain important business critical data.
>> - A single photo file has a size between 1 and 10 MB.
>> - The photos need to be stored over several years (10-30 years).
>> - The data store should support replication over several servers.
>> - A checksum-concept is needed to guarantee the data integrity of all
>> files over a long period of time.
>> - To write and read the files a Rest API is preferred.
>>
>> So far Hadoop seems to be absolutely the perfect solution. But my last
>> requirement seems to throw Hadoop out of the race.
>>
>> - The photos need to be readable with very short latency from an external
>> enterprise application
>>
>> With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems
>> that most of the Hadoop experts advise against this usage if the size of my
>> data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
>>
>> I think I understood the concepts of HAR or sequential files.
>> But if I pack, for example, my files together in a large file of many
>> Gigabytes it is impossible to access one single photo from the Hadoop
>> repository in a reasonable time. It makes no sense in my eyes to pack
>> thousands of files into a large file just so that Hadoop jobs can handle it
>> better. To simply access a single file from a web interface - as in my case
>> - it seems to be all counterproductive.
>>
>> So my question is: Is Hadoop only feasible to archive large Web-server
>> log files and not designed to handle big archives of small files with also
>> business critical data?
>>
>>
>> Thanks for your advice in advance.
>>
>> Ralph
>> --
>>
>>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org For additional
> commands, e-mail: user-help@hadoop.apache.org
--
Hayati Gonultas
Re: Is Hadoop basically not suitable for a photo archive?
Posted by Alexey Eremihin <a....@corp.badoo.com.INVALID>.
Hi Ralph,
In general Hadoop is able to store such data. And even Har archives can be
used with conjunction with WebHDFS (by passing offset and limit
attributes). What are your reading requirements? FS meta data are not
distributed and reading the data is limited by the HDFS NameNode server
performance. So if you would like to download files with high RPS that
would not work well.
On Monday, September 4, 2017, Ralph Soika <ra...@imixs.com> wrote:
> Hi,
>
> I know that the issue around the small-file problem was asked frequently,
> not only in this mailing list.
> I also have read already some books about Haddoop and I also started to
> work with Hadoop. But still I did not really understand if Hadoop is the
> right choice for my goals.
>
> To simplify my problem domain I would like to use the use case of a photo
> archive:
>
> - An external application produces about 10 million photos in one year.
> The files contain important business critical data.
> - A single photo file has a size between 1 and 10 MB.
> - The photos need to be stored over several years (10-30 years).
> - The data store should support replication over several servers.
> - A checksum-concept is needed to guarantee the data integrity of all
> files over a long period of time.
> - To write and read the files a Rest API is preferred.
>
> So far Hadoop seems to be absolutely the perfect solution. But my last
> requirement seems to throw Hadoop out of the race.
>
> - The photos need to be readable with very short latency from an external
> enterprise application
>
> With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems
> that most of the Hadoop experts advise against this usage if the size of my
> data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
>
> I think I understood the concepts of HAR or sequential files.
> But if I pack, for example, my files together in a large file of many
> Gigabytes it is impossible to access one single photo from the Hadoop
> repository in a reasonable time. It makes no sense in my eyes to pack
> thousands of files into a large file just so that Hadoop jobs can handle it
> better. To simply access a single file from a web interface - as in my case
> - it seems to be all counterproductive.
>
> So my question is: Is Hadoop only feasible to archive large Web-server log
> files and not designed to handle big archives of small files with also
> business critical data?
>
>
> Thanks for your advice in advance.
>
> Ralph
> --
>
>
RE: Is Hadoop basically not suitable for a photo archive?
Posted by "Zheng, Kai" <ka...@intel.com>.
Looks like HBase MOB should be mentioned, since the feature was definitely introduced with photo files/objects in mind.
Regards,
Kai
From: Grant Overby [mailto:grant.overby@gmail.com]
Sent: Thursday, September 07, 2017 3:05 AM
To: Ralph Soika <ra...@imixs.com>
Cc: user@hadoop.apache.org
Subject: Re: Is Hadoop basically not suitable for a photo archive?
I'm late to the party, and this isn't a hadoop solution, but apparently Cassandra is pretty good at this.
https://medium.com/walmartlabs/building-object-store-storing-images-in-cassandra-walmart-scale-a6b9c02af593
On Wed, Sep 6, 2017 at 2:48 PM, Ralph Soika <ra...@imixs.com>> wrote:
Hi
I want to thank you all for your answers and your good ideas how to solve the hadoop "small-file-problem".
Now I would like to briefly summarize your answers and suggested solutions. First of all I describe once again my general use case:
* An external enterprise application need to store small photo files in unsteady intervals into a clustered big data storage.
* Users need to read the files through the web interface of the enterprise application also in unsteady intervals.
* The solution need to guarantee the data integrity of all files over a long period of time.
* To write and read the files a Rest API is preferred.
1) Multiple small-files in one sequence file:
Packing multiple small files in one sequence file is a possible solution even if it is hard to implement. If the files are attached by the enterprise application in unsteady intervals (as in my case), the enterprise application need to be aware of the latest filesize of the sequence file and need to compute the correct offset. The offset and the size of the photo file is needed to access a single photo file later. E.g. via WebHDFS:
http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=<LONG>]&length=<LONG<http://%3cHOST%3e:%3cPORT%3e/webhdfs/v1/%3cPATH%3e?op=OPEN&offset=%3cLONG%3e%5d&length=%3cLONG>>]
If multiple threads try to append data in parallel, the problem becomes more and more complex. But yes - this could be a possible solution.
2) Multiple small-files in a Hadoop Archive (HAR):
Another solution is to pack the small files in a Hadoop Archive (HAR) file. But this solution is even more difficult to implement in my case. As I explained, the enterprise application write data in unsteady intervals. This means that the archive job need to be decoupled from the enterprise application. For example a scheduler could archive and delete files older than one day on a daily basis. This would reduce the number of small files significant. The problem here is, that the enterprise application need to be aware of the new location of a single photo file. To access a 'packed' photo file, the Offset and Size in the HAR file need to be transferred back to the enterprise application. As a result of this solution, the complexity of the overall system increases unreasonably. To decouple things, the scheduler could create a kind of index file for each new created HAR file. The index could be used by the enterprise application to lookup the file-path, offset and size. But as a single photo file can now be either stored still as a small-file or already as a part of a HAR file, the access method need to be implemented very tricky.
OK, the solution is possible, but the sequence file solution seems to be much easier.
3) The object store "openstack swift":
It seems that the object store "openstack swift" is solving the small-file problem much better. It is certainly worth following this approach. However, since I am basically convinced of Hadoop, I will not make a fundamental change in architecture for now.
4) Intel-bigdata/SSM:
The "Transparent Small Files Support" form the Intel-Bigdata project is an interesting approach and I believe this would solve my problems at all. But I fear it is to early to start here.
5) HDFS-7240 or Ozone:
HDFS-7240 or Ozone looks very very promising. It looks to me that Ozone is the missing piece in the Hadoop project. Although it is not yet ready for use in production, I will follow this project.
6) mapR-fs:
mapR-fs could be an alternative, but I do not consider it here.
My conclusion:
So for my use case in short-term it seems to be the best solution to start with the sequence file approach. In the intermediate-term I will see if I can adapt my solution to the Hadoop Ozone project. In the long term I will probably support both approaches.
Since my solution is part of the open source project Imixs-Workflow, I will publish my solution as well on GitHub.
So once again - thanks a lot for your help.
Ralph
On 04.09.2017 19:03, Ralph Soika wrote:
Hi,
I know that the issue around the small-file problem was asked frequently, not only in this mailing list.
I also have read already some books about Haddoop and I also started to work with Hadoop. But still I did not really understand if Hadoop is the right choice for my goals.
To simplify my problem domain I would like to use the use case of a photo archive:
- An external application produces about 10 million photos in one year. The files contain important business critical data.
- A single photo file has a size between 1 and 10 MB.
- The photos need to be stored over several years (10-30 years).
- The data store should support replication over several servers.
- A checksum-concept is needed to guarantee the data integrity of all files over a long period of time.
- To write and read the files a Rest API is preferred.
So far Hadoop seems to be absolutely the perfect solution. But my last requirement seems to throw Hadoop out of the race.
- The photos need to be readable with very short latency from an external enterprise application
With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems that most of the Hadoop experts advise against this usage if the size of my data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
I think I understood the concepts of HAR or sequential files.
But if I pack, for example, my files together in a large file of many Gigabytes it is impossible to access one single photo from the Hadoop repository in a reasonable time. It makes no sense in my eyes to pack thousands of files into a large file just so that Hadoop jobs can handle it better. To simply access a single file from a web interface - as in my case - it seems to be all counterproductive.
So my question is: Is Hadoop only feasible to archive large Web-server log files and not designed to handle big archives of small files with also business critical data?
Thanks for your advice in advance.
Ralph
--
--
Imixs...extends the way people work together
We are an open source company, read more at: www.imixs.org<http://www.imixs.org>
________________________________
Imixs Software Solutions GmbH
Agnes-Pockels-Bogen 1, 80992 München
Web: www.imixs.com<http://www.imixs.com>
Office: +49 (0)89-452136 16<tel:+49%2089%2045213616> Mobil: +49-177-4128245<tel:+49%20177%204128245>
Registergericht: Amtsgericht Muenchen, HRB 136045
Geschaeftsfuehrer: Gaby Heinle u. Ralph Soika
Re: Is Hadoop basically not suitable for a photo archive?
Posted by Grant Overby <gr...@gmail.com>.
I'm late to the party, and this isn't a hadoop solution, but apparently
Cassandra is pretty good at this.
https://medium.com/walmartlabs/building-object-store-storing-images-in-cassandra-walmart-scale-a6b9c02af593
On Wed, Sep 6, 2017 at 2:48 PM, Ralph Soika <ra...@imixs.com> wrote:
> Hi
>
> I want to thank you all for your answers and your good ideas how to solve
> the hadoop "small-file-problem".
>
> Now I would like to briefly summarize your answers and suggested
> solutions. First of all I describe once again my general use case:
>
>
> - An external enterprise application need to store small photo files
> in unsteady intervals into a clustered big data storage.
> - Users need to read the files through the web interface of the
> enterprise application also in unsteady intervals.
> - The solution need to guarantee the data integrity of all files over
> a long period of time.
> - To write and read the files a Rest API is preferred.
>
>
>
> *1) Multiple small-files in one sequence file:*
>
> Packing multiple small files in one sequence file is a possible solution
> even if it is hard to implement. If the files are attached by the
> enterprise application in unsteady intervals (as in my case), the
> enterprise application need to be aware of the latest filesize of the
> sequence file and need to compute the correct offset. The offset and the
> size of the photo file is needed to access a single photo file later. E.g.
> via WebHDFS:
>
> http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=<
> LONG>]&length=<LONG>]
>
> If multiple threads try to append data in parallel, the problem becomes
> more and more complex. But yes - this could be a possible solution.
>
>
> *2) **Multiple small-files in a Hadoop Archive (HAR):*
>
> Another solution is to pack the small files in a Hadoop Archive (HAR)
> file. But this solution is even more difficult to implement in my case. As
> I explained, the enterprise application write data in unsteady intervals.
> This means that the archive job need to be decoupled from the enterprise
> application. For example a scheduler could archive and delete files older
> than one day on a daily basis. This would reduce the number of small files
> significant. The problem here is, that the enterprise application need to
> be aware of the new location of a single photo file. To access a 'packed'
> photo file, the Offset and Size in the HAR file need to be transferred back
> to the enterprise application. As a result of this solution, the complexity
> of the overall system increases unreasonably. To decouple things, the
> scheduler could create a kind of index file for each new created HAR file.
> The index could be used by the enterprise application to lookup the
> file-path, offset and size. But as a single photo file can now be either
> stored still as a small-file or already as a part of a HAR file, the access
> method need to be implemented very tricky.
> OK, the solution is possible, but the sequence file solution seems to be
> much easier.
>
>
>
> *3) The object store "openstack swift":*
>
> It seems that the object store "openstack swift" is solving the small-file
> problem much better. It is certainly worth following this approach.
> However, since I am basically convinced of Hadoop, I will not make a
> fundamental change in architecture for now.
>
>
> *4) Intel-bigdata/SSM:*
>
> The "Transparent Small Files Support" form the Intel-Bigdata project is an
> interesting approach and I believe this would solve my problems at all. But
> I fear it is to early to start here.
>
>
>
> *5) HDFS-7240 or Ozone:*
>
> HDFS-7240 or Ozone looks very very promising. It looks to me that Ozone is
> the missing piece in the Hadoop project. Although it is not yet ready for
> use in production, I will follow this project.
>
>
> *6) mapR-fs:*
> mapR-fs could be an alternative, but I do not consider it here.
>
>
>
>
> *My conclusion: *
>
> So for my use case in short-term it seems to be the best solution to start
> with the sequence file approach. In the intermediate-term I will see if I
> can adapt my solution to the Hadoop Ozone project. In the long term I will
> probably support both approaches.
> Since my solution is part of the open source project Imixs-Workflow, I
> will publish my solution as well on GitHub.
>
>
> So once again - thanks a lot for your help.
>
> Ralph
>
>
>
>
> On 04.09.2017 19:03, Ralph Soika wrote:
>
> Hi,
>
> I know that the issue around the small-file problem was asked frequently,
> not only in this mailing list.
> I also have read already some books about Haddoop and I also started to
> work with Hadoop. But still I did not really understand if Hadoop is the
> right choice for my goals.
>
> To simplify my problem domain I would like to use the use case of a photo
> archive:
>
> - An external application produces about 10 million photos in one year.
> The files contain important business critical data.
> - A single photo file has a size between 1 and 10 MB.
> - The photos need to be stored over several years (10-30 years).
> - The data store should support replication over several servers.
> - A checksum-concept is needed to guarantee the data integrity of all
> files over a long period of time.
> - To write and read the files a Rest API is preferred.
>
> So far Hadoop seems to be absolutely the perfect solution. But my last
> requirement seems to throw Hadoop out of the race.
>
> - The photos need to be readable with very short latency from an external
> enterprise application
>
> With Hadoop HDFS and the Web Proxy everything seems perfect. But it seems
> that most of the Hadoop experts advise against this usage if the size of my
> data files (1-10 MB) are well below the Hadoop block size of 64 or 128 MB.
>
> I think I understood the concepts of HAR or sequential files.
> But if I pack, for example, my files together in a large file of many
> Gigabytes it is impossible to access one single photo from the Hadoop
> repository in a reasonable time. It makes no sense in my eyes to pack
> thousands of files into a large file just so that Hadoop jobs can handle it
> better. To simply access a single file from a web interface - as in my case
> - it seems to be all counterproductive.
>
> So my question is: Is Hadoop only feasible to archive large Web-server log
> files and not designed to handle big archives of small files with also
> business critical data?
>
>
> Thanks for your advice in advance.
>
> Ralph
> --
>
>
> --
> *Imixs*...extends the way people work together
> We are an open source company, read more at: www.imixs.org
> ------------------------------
> Imixs Software Solutions GmbH
> Agnes-Pockels-Bogen 1, 80992 München
> *Web:* www.imixs.com
> *Office:* +49 (0)89-452136 16 <+49%2089%2045213616> *Mobil:*
> +49-177-4128245 <+49%20177%204128245>
> Registergericht: Amtsgericht Muenchen, HRB 136045
> Geschaeftsfuehrer: Gaby Heinle u. Ralph Soika
>
Re: Is Hadoop basically not suitable for a photo archive?
Posted by Ralph Soika <ra...@imixs.com>.
Hi
I want to thank you all for your answers and your good ideas how to
solve the hadoop "small-file-problem".
Now I would like to briefly summarize your answers and suggested
solutions. First of all I describe once again my general use case:
* An external enterprise application need to store small photo files
in unsteady intervals into a clustered big data storage.
* Users need to read the files through the web interface of the
enterprise application also in unsteady intervals.
* The solution need to guarantee the data integrity of all files over
a long period of time.
* To write and read the files a Rest API is preferred.
_1) Multiple small-files in one sequence file:_
Packing multiple small files in one sequence file is a possible solution
even if it is hard to implement. If the files are attached by the
enterprise application in unsteady intervals (as in my case), the
enterprise application need to be aware of the latest filesize of the
sequence file and need to compute the correct offset. The offset and the
size of the photo file is needed to access a single photo file later.
E.g. via WebHDFS:
http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN&offset=<LONG>]&length=<LONG>]
If multiple threads try to append data in parallel, the problem becomes
more and more complex. But yes - this could be a possible solution.
_2) __Multiple small-files in a Hadoop Archive (HAR):_
Another solution is to pack the small files in a Hadoop Archive (HAR)
file. But this solution is even more difficult to implement in my case.
As I explained, the enterprise application write data in unsteady
intervals. This means that the archive job need to be decoupled from the
enterprise application. For example a scheduler could archive and delete
files older than one day on a daily basis. This would reduce the number
of small files significant. The problem here is, that the enterprise
application need to be aware of the new location of a single photo file.
To access a 'packed' photo file, the Offset and Size in the HAR file
need to be transferred back to the enterprise application. As a result
of this solution, the complexity of the overall system increases
unreasonably. To decouple things, the scheduler could create a kind of
index file for each new created HAR file. The index could be used by the
enterprise application to lookup the file-path, offset and size. But as
a single photo file can now be either stored still as a small-file or
already as a part of a HAR file, the access method need to be
implemented very tricky.
OK, the solution is possible, but the sequence file solution seems to be
much easier.
_3) The object store "openstack swift":_
It seems that the object store "openstack swift" is solving the
small-file problem much better. It is certainly worth following this
approach. However, since I am basically convinced of Hadoop, I will not
make a fundamental change in architecture for now.
_4) Intel-bigdata/SSM:_
The "Transparent Small Files Support" form the Intel-Bigdata project is
an interesting approach and I believe this would solve my problems at
all. But I fear it is to early to start here.
_5) HDFS-7240 or Ozone:_
HDFS-7240 or Ozone looks very very promising. It looks to me that Ozone
is the missing piece in the Hadoop project. Although it is not yet ready
for use in production, I will follow this project.
_6) mapR-fs:_
mapR-fs could be an alternative, but I do not consider it here.
_My conclusion: _
So for my use case in short-term it seems to be the best solution to
start with the sequence file approach. In the intermediate-term I will
see if I can adapt my solution to the Hadoop Ozone project. In the long
term I will probably support both approaches.
Since my solution is part of the open source project Imixs-Workflow, I
will publish my solution as well on GitHub.
So once again - thanks a lot for your help.
Ralph
On 04.09.2017 19:03, Ralph Soika wrote:
>
> Hi,
>
> I know that the issue around the small-file problem was asked
> frequently, not only in this mailing list.
> I also have read already some books about Haddoop and I also started
> to work with Hadoop. But still I did not really understand if Hadoop
> is the right choice for my goals.
>
> To simplify my problem domain I would like to use the use case of a
> photo archive:
>
> - An external application produces about 10 million photos in one
> year. The files contain important business critical data.
> - A single photo file has a size between 1 and 10 MB.
> - The photos need to be stored over several years (10-30 years).
> - The data store should support replication over several servers.
> - A checksum-concept is needed to guarantee the data integrity of all
> files over a long period of time.
> - To write and read the files a Rest API is preferred.
>
> So far Hadoop seems to be absolutely the perfect solution. But my last
> requirement seems to throw Hadoop out of the race.
>
> - The photos need to be readable with very short latency from an
> external enterprise application
>
> With Hadoop HDFS and the Web Proxy everything seems perfect. But it
> seems that most of the Hadoop experts advise against this usage if the
> size of my data files (1-10 MB) are well below the Hadoop block size
> of 64 or 128 MB.
>
> I think I understood the concepts of HAR or sequential files.
> But if I pack, for example, my files together in a large file of many
> Gigabytes it is impossible to access one single photo from the Hadoop
> repository in a reasonable time. It makes no sense in my eyes to pack
> thousands of files into a large file just so that Hadoop jobs can
> handle it better. To simply access a single file from a web interface
> - as in my case - it seems to be all counterproductive.
>
> So my question is: Is Hadoop only feasible to archive large Web-server
> log files and not designed to handle big archives of small files with
> also business critical data?
>
>
> Thanks for your advice in advance.
>
> Ralph
>
> --
>
--
*Imixs*...extends the way people work together
We are an open source company, read more at: www.imixs.org
<http://www.imixs.org>
------------------------------------------------------------------------
Imixs Software Solutions GmbH
Agnes-Pockels-Bogen 1, 80992 München
*Web:* www.imixs.com <http://www.imixs.com>
*Office:* +49 (0)89-452136 16 *Mobil:* +49-177-4128245
Registergericht: Amtsgericht Muenchen, HRB 136045
Geschaeftsfuehrer: Gaby Heinle u. Ralph Soika