You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Thomas Weise (JIRA)" <ji...@apache.org> on 2015/12/28 00:57:49 UTC
[jira] [Reopened] (APEXCORE-2) Enable HDFS write on streams as an
attribute
[ https://issues.apache.org/jira/browse/APEXCORE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise reopened APEXCORE-2:
---------------------------------
> Enable HDFS write on streams as an attribute
> --------------------------------------------
>
> Key: APEXCORE-2
> URL: https://issues.apache.org/jira/browse/APEXCORE-2
> Project: Apache Apex Core
> Issue Type: Task
> Reporter: Gavin
> Assignee: Chetan Narsude
> Priority: Critical
>
> On a stream, enable spooling the streams to HDFS. This needs the following to be specified
> - Physical partitions: Since partitions are decided via input port, putting partitions on stream (written to by output ports) requires special handling
> - Write policy (directory, partitionining to be consumed by consumers, hive?, hcatalog, ...)
> - Ability to ask for partitions to be derived from bandwidth. Assume one physical HDFS write can only do 30 MB/sec. User should be able to say "derive the number of HDFS writers", or "use N writers and grow them if throughput goes up"
> Amol
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)