Fw: Data copy from HDFS to MinIO regularly

[Arif Kamil Yilmaz <a_...@yahoo.com.INVALID>]

 
Subject: Data copy from HDFS to MinIO regularly
 
Hello Team,

There is an application that was developed a long time ago, and this application processes 10GB of binary data per hour using MapReduce and generates 100GB of data, which is then written to the HDFS file system.

My goal is to move a portion of the processed data (approximately 25%) to a MinIO cluster that I plan to use as new object storage. I want this operation to be repeated every time new data is added to the HDFS cluster.

What kind of solution would you suggest to complete this task? Additionally, I would like to remind you that I have requirements related to monitoring the pipeline I am developing.

Thank you.

  

Re: Fw: Data copy from HDFS to MinIO regularly

[Steve Loughran <st...@cloudera.com.INVALID>]

distcp can upload a directory tree of changed files, for cloud storage it
looks for the different in file timestamps

Otherwise, the HDFS namenode has an log4j audit logger:
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit

This prints out all namenode filesystem operations in a structured format
(which is consistent across releases)...you can read that and use whatever
copy command you want to use to upload it.
https://stackoverflow.com/questions/44533589/hdfs-audit-logs-format-and-explanation

AFAIK There's no tool which takes a list of changed files and uploads them;
I've played with using spark for this but it's not easy.

the cloud store app we use for diagnostics has a 'cloudup` command to
upload a directory tree to cloud storage (or any hadoop filesystem to
filesystem)
https://github.com/steveloughran/cloudstore
https://github.com/steveloughran/cloudstore/blob/main/src/main/site/cloudup.md

That's a single process but it is aggressive, optimised for cloud storage
with rate limiting by shard
https://github.com/steveloughran/cloudstore/blob/main/src/main/site/cloudup.md

you could probably modify that to take a list of source + destination paths
to build its upload list and let it do the work.

You would have to do that on your own you, I'm afraid -through a PR of your
changes would be welcome, especially with tests.

steve


On Wed, 12 Mar 2025 at 04:35, Arif Kamil Yilmaz
<a_...@yahoo.com.invalid> wrote:

>
> Subject: Data copy from HDFS to MinIO regularly
>
> Hello Team,
>
> There is an application that was developed a long time ago, and this
> application processes 10GB of binary data per hour using MapReduce and
> generates 100GB of data, which is then written to the HDFS file system.
>
> My goal is to move a portion of the processed data (approximately 25%) to
> a MinIO cluster that I plan to use as new object storage. I want this
> operation to be repeated every time new data is added to the HDFS cluster.
>
> What kind of solution would you suggest to complete this task?
> Additionally, I would like to remind you that I have requirements related
> to monitoring the pipeline I am developing.
>
> Thank you.
>
>