You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Віталій Тимчишин <ti...@gmail.com> on 2011/07/17 18:54:00 UTC

Re: Hadoop Production Issue

2011/7/16 jagaran das <ja...@yahoo.co.in>

> Hi,
>
> Due to requirements in our current production CDH3 cluster we need to copy
> around 11520 small size files (Total Size 12 GB) to the cluster for one
> application.
> Like this we have 20 applications that would run in parallel
>
> So one set would have 11520 files of total size 12 GB
> Like this we would have 15 sets in parallel,
>
> We have a total SLA for the pipeline from copy to pig aggregation to copy
> to local and sql load is 15 mins.
>
>
Have you tried to use HARs?
-- 
Best regards,
 Vitalii Tymchyshyn