[jira] (NIFI-12923) PutHDFS to support appending avro data

["Joe Witt (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-12923 ]


    Joe Witt deleted comment on NIFI-12923:
    ---------------------------------

was (Author: joewitt):
[~balazsgerner][~mattyb149] I am reverting the commit for this test change as the two tests are unstable and need review/resolution.


{noformat}
[ERROR] Tests run: 20, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.478 s <<< FAILURE! -- in org.apache.nifi.processors.hadoop.PutHDFSTest
[ERROR] org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeNewFileCreated -- Time elapsed: 0.005 s <<< ERROR!
java.io.FileNotFoundException: src/test/resources/testdata-avro/input.avro (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106)
	at org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeNewFileCreated(PutHDFSTest.java:281)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[ERROR] org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeWhenTargetFileAlreadyExists -- Time elapsed: 0.009 s <<< ERROR!
java.io.FileNotFoundException: src/test/resources/testdata-avro/input.avro (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106)
	at org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeWhenTargetFileAlreadyExists(PutHDFSTest.java:308)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

{noformat}


> PutHDFS to support appending avro data
> --------------------------------------
>
>                 Key: NIFI-12923
>                 URL: https://issues.apache.org/jira/browse/NIFI-12923
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Balázs Gerner
>            Assignee: Balázs Gerner
>            Priority: Major
>             Fix For: 2.0.0-M3, 1.26.0
>
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The goal of this ticket is to extend the PutHDFS processor with the ability to append avro records. The processor already provides an option to set 'append' as conflict resolution strategy, but that does not work correctly in case of avro files, because the serialized avro file cannot be deserialized again (because the binary content is invalid).
> Some notes about the implementation:
>  * The user needs to explicitly select avro as file format and append as conflict resolution mode to enable 'avro append' mode, otherwise regular append mode will work just as before. There is no auto detection of mimetype for the incoming flowfile.
>  * The records of the incoming flowfile and the ones in the existing avro file need to conform to the same avro schema, otherwise the append operation fails with incompatible schema.
>  * The 'avro append' mode should only work when compression type is set to 'none', if any other compression type is selected in 'avro append' mode the user should get a validation error.
> The changes will have to be added to *support/nifi-1.x* branch also.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)