You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Jozef Vilcek (JIRA)" <ji...@apache.org> on 2018/06/28 10:43:00 UTC
[jira] [Created] (BEAM-4672) Dynamic destination write to HDFS
fails with IntervalWindow in shard template
Jozef Vilcek created BEAM-4672:
----------------------------------
Summary: Dynamic destination write to HDFS fails with IntervalWindow in shard template
Key: BEAM-4672
URL: https://issues.apache.org/jira/browse/BEAM-4672
Project: Beam
Issue Type: Bug
Components: io-java-hadoop
Affects Versions: 2.4.0
Reporter: Jozef Vilcek
Assignee: Chamikara Jayalath
I am doing a windowed write with dynamic destinations and it is failing with error message :
{noformat}
java.lang.IllegalArgumentException: Expect srcResourceIds and destResourceIds have the same scheme, but received hdfs, window-1970-01-01T00{noformat}
I am writing from IntervalWindow, for which target file name created by DefaultFilenamePolicy is going to look something like this:
{noformat}
window-1970-01-01T00:00:00.000Z-1970-01-01T01:00:00.000Z-pane-0-last-00000-of-00001.avro{noformat}
When resolving the path, HadoopResourceId is used and this one is backed by java URI. The filename like above is parsed incorrectly because of ':' character introduced by window-to-string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)