You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Miguel Costa (Jira)" <ji...@apache.org> on 2023/01/05 17:23:00 UTC
[jira] [Created] (AVRO-3699) avro-tools dependency on hadoop
Miguel Costa created AVRO-3699:
----------------------------------
Summary: avro-tools dependency on hadoop
Key: AVRO-3699
URL: https://issues.apache.org/jira/browse/AVRO-3699
Project: Apache Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.11.1
Reporter: Miguel Costa
Fix For: 1.12.0
Hi all, this is not something that hinders the usage of the avro tool but I don't understand why this dependency exists.
Seems to be more for convenience than anything but for example when I ran the Avro-tool in java I always get this warnings that are a bit annoying.
{code:java}
23/01/05 18:21:24 WARN util.Shell: Did not find winutils.exe: {}
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:547)
at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:568)
at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:591)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:688)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3741)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3736)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.avro.tool.Util.openFromFS(Util.java:88)
at org.apache.avro.tool.Util.fileOrStdin(Util.java:64)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:87)
at org.apache.avro.tool.Main.run(Main.java:67)
at org.apache.avro.tool.Main.main(Main.java:56)
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:467)
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:438)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:515)
... 13 more
23/01/05 18:21:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Output:
{"name":"r","age":{"int":906},"gender":"MALE","address":{"zipcode":631}}
{"name":"oYBuz","age":{"int":255},"gender":"FEMALE","address":{"zipcode":690}}
{"name":"xKMgdHyLw","age":null,"gender":"FEMALE","address":{"zipcode":304}}
{"name":"MPPsYun","age":null,"gender":"MALE","address":{"zipcode":875}}
{"name":"XTrq","age":{"int":167},"gender":"MALE","address":{"zipcode":973}}
{"name":"g","age":null,"gender":"FEMALE","address":{"zipcode":351}}
{"name":"ynx","age":null,"gender":"FEMALE","address":{"zipcode":525}}
{"name":"xPFZ","age":null,"gender":"FEMALE","address":{"zipcode":921}}
{"name":"bRHCLEwdglb","age":{"int":541},"gender":"MALE","address":{"zipcode":123}}
{"name":"UVcDVhxpyCziyBSiRasp","age":null,"gender":"FEMALE","address":{"zipcode":53}}
{code}
It would be great if this could be improved in a future.
It might just be that there are some hard requirements in other parts of the tool that require this, so in that case this issue makes no sense.
But I did try to use the util.logging or log4j config files to ignore this warnings but I was not so lucky.
something like:
{code:java}
java -Djava.util.logging.config.file=utillog.properties -jar avro-tools-1.11.1.jar getschema avro/person-10.avro{code}
{code:java}
java -Dlog4j.configurationFile=log4j.properties -jar avro-tools-1.11.1.jar getschema avro/person-10.avro{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)