You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Akira AJISAKA (JIRA)" <ji...@apache.org> on 2015/12/18 06:03:47 UTC
[jira] [Commented] (HADOOP-12657) Add a option to skip newline on
empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HADOOP-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063466#comment-15063466 ]
Akira AJISAKA commented on HADOOP-12657:
----------------------------------------
and thanks [~jfilipiak] for reviewing the patches!
> Add a option to skip newline on empty files with getMerge -nl
> -------------------------------------------------------------
>
> Key: HADOOP-12657
> URL: https://issues.apache.org/jira/browse/HADOOP-12657
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 2.6.0, 2.7.1
> Reporter: Jan Filipiak
> Assignee: Kanaka Kumar Avvaru
> Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch, HDFS-8836-03.patch, HDFS-8836-04.patch, HDFS-8836-05.patch, HDFS-8836-06.patch, HDFS-8836-07.patch
>
>
> Hello everyone,
> I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards.
> In this situation it is maybe good to have another argument that allows skipping empty files.
> Thing one could try to implement this feature:
> The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
> return the number of bytes copied which would be convenient as one could
> skip append the new line when 0 bytes where copied or one would check the file size before.
> I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)