You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2019/02/04 17:33:00 UTC
[jira] [Created] (MAPREDUCE-7182) MapReduce input format/record
readers to support S3 select queries
Steve Loughran created MAPREDUCE-7182:
-----------------------------------------
Summary: MapReduce input format/record readers to support S3 select queries
Key: MAPREDUCE-7182
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7182
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 3.3.0
Reporter: Steve Loughran
HADOOP-15229 adds S3 select through the (new) async openFile API, but the classic RecordReader &c can't handle it because
# the files are shorter than they are in a getFileStatus, and the readers assume that an EOFException is an error in that situation
# everything assumes plain text is splitable
# if a file has a gz extension, the gunzip codec should be used. So breaks transcoded/uncompressed data
to handle s3 select data sources we need to be able to address them, either through changes to the existing code (danger?) or some new readers
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org