You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Chris D'Urso (JIRA)" <ji...@apache.org> on 2014/03/25 18:08:23 UTC
[jira] [Updated] (AVRO-1486) avro-tools-1.7.6.jar fromtext to
stdout fails because status also printed to stdout
[ https://issues.apache.org/jira/browse/AVRO-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris D'Urso updated AVRO-1486:
-------------------------------
Description:
Apparently unwanted status information is being printed out to standard output. This interferes with the '-' option when attempting to direct aver encoded binary data to a pipe. A possible workaround is to use a named pipe instead, but really the program should either be quiet with its status output at least in the case where stdout is the file handle directed by the '-' option.
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one 1
two 2
three 3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be equivalent is not so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and this is because its been written to stdout as well as the true payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155
was:
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one 1
two 2
three 3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be equivalent is not so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and this is because its been written to stdout as well as the true payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
one 1
two 2
three 3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155
> avro-tools-1.7.6.jar fromtext to stdout fails because status also printed to stdout
> ------------------------------------------------------------------------------------
>
> Key: AVRO-1486
> URL: https://issues.apache.org/jira/browse/AVRO-1486
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.7.6
> Reporter: Chris D'Urso
>
> Apparently unwanted status information is being printed out to standard output. This interferes with the '-' option when attempting to direct aver encoded binary data to a pipe. A possible workaround is to use a named pipe instead, but really the program should either be quiet with its status output at least in the case where stdout is the file handle directed by the '-' option.
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation of test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - test.txt.avro
> Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
> WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Writing line = 5
> Writing line = 5
> Writing line = 7
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
> -rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> f@5:B
> one 1
> two 2
> three 3
> f@5:B
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
> 104 test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # now what should be equivalent is not so
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar fromtext --codec snappy - - > test2.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and this is because its been written to stdout as well as the true payload
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> gD?Writing line = 5
> Writing line = 5
> Writing line = 7
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
> one 1
> two 2
> three 3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar totext - -
> Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
> at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
> at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
> at org.apache.avro.tool.Main.run(Main.java:84)
> at org.apache.avro.tool.Main.main(Main.java:73)
> Caused by: java.io.IOException: Invalid sync!
> at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
> at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
> ... 3 more
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
> 155
--
This message was sent by Atlassian JIRA
(v6.2#6252)