Initial delay and high memory footprint in gobblin distcp

[Rohit Kalhans <ro...@gmail.com>]

Hello,

We are using gobblin to ingest data from a remote hdfc cluster to a local
one. Both clusters are in same geolocation. How ever when we start I hating
data we observe two behaviour.

1 The JVM  heap usage is erratic. Near the beginning of the job start we
see frequent GC on the driver jvm. The memory usage is also really high. We
have around 56000 files and the size is 1TB

2 The initial time spent by the job on the driver box is nearly 6times of
the map job duration. This mean we are spending almost 6times rounding up
the files  compare to actual copy.

We would like to know if there is a way we could avoid these two pitfalls
while using distcp-NG.

We are ok if the data is inconsistent at the dataset level as long as
individual files are copied correctly.

Another thing, we see there are a few parameters which are used in the code
but I could not find their reference in the docs. E.g.
gobblin.prioritization.maxCopy.copyEntities
Gobblin.copy.max.concurrent.listing.services
Gobblin.copy.binpacking.Maxworkunitperbin
Gobblin.copy.binpacking.MaxsizePerBin
Gobblin.copy.abortonsinglefailure


Is there a better way to understand these other than looking into the code
?

-- 
Cheerio!

*Rohit*