You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Amit Katti (JIRA)" <ji...@apache.org> on 2014/07/21 22:32:39 UTC
[jira] [Comment Edited] (DRILL-1075) can not create hdfs as
connection type in storage engine : server throws http 500 error
[ https://issues.apache.org/jira/browse/DRILL-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069228#comment-14069228 ]
Amit Katti edited comment on DRILL-1075 at 7/21/14 8:31 PM:
------------------------------------------------------------
I just finished testing DRILL on vanilla Hadoop and CDH4 and CDH5 and it is working successfully.
For Drill to work on Hadoop, you need to provide the connection as "connection" : "hdfs://10.10.30.156:8020/"
The storage plugin for dfs is below
{code}
{
"type" : "file",
"enabled" : true,
"connection" : "hdfs://10.10.30.156:8020/",
"workspaces" : {
"root" : {
"location" : "/user/root/drill",
"writable" : true,
"storageformat" : "null"
},
"tmp" : {
"location" : "/tmp",
"writable" : true,
"storageformat" : "csv"
},
"drillTestDir" : {
"location" : "/drill/testdata/",
"writable" : false,
"storageformat" : "parquet"
}
},
"formats" : {
"psv" : {
"type" : "text",
"extensions" : [ "tbl" ],
"delimiter" : "|"
},
"csv" : {
"type" : "text",
"extensions" : [ "csv" ],
"delimiter" : ","
},
"tsv" : {
"type" : "text",
"extensions" : [ "tsv" ],
"delimiter" : "t"
},
"parquet" : {
"type" : "parquet"
},
"json" : {
"type" : "json"
}
}
}
{code}
For this to work you need to have the below mentioned jars in the Drill Classpath (location might change depending on installation):
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar
There is a separate Drill Jira open for ensuring this is configured correctly at DRILL-1160
https://issues.apache.org/jira/browse/DRILL-1160
was (Author: amitskatti):
I just finished testing DRILL on vanilla Hadoop and CDH4 and CDH5 and it is working successfully.
For Drill to work on Hadoop, you need to provide the connection as "connection" : "hdfs://10.10.30.156:8020/"
The storage plugin for dfs is below
{code}
{
"type" : "file",
"enabled" : true,
"connection" : "hdfs://10.10.30.156:8020/",
"workspaces" : {
"root" : {
"location" : "/user/root/drill",
"writable" : true,
"storageformat" : "null"
},
"tmp" : {
"location" : "/tmp",
"writable" : true,
"storageformat" : "csv"
},
"drillTestDir" : {
"location" : "/drill/testdata/",
"writable" : false,
"storageformat" : "parquet"
}
},
"formats" : {
"psv" : {
"type" : "text",
"extensions" : [ "tbl" ],
"delimiter" : "|"
},
"csv" : {
"type" : "text",
"extensions" : [ "csv" ],
"delimiter" : ","
},
"tsv" : {
"type" : "text",
"extensions" : [ "tsv" ],
"delimiter" : "t"
},
"parquet" : {
"type" : "parquet"
},
"json" : {
"type" : "json"
}
}
}
{code}
For this to work you need to have the below mentioned jars in the Drill Classpath:
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-annotations-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-auth-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop/hadoop-common-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.7.0.jar
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar
There is a separate Drill Jira open for ensuring this is configured correctly at DRILL-1160
https://issues.apache.org/jira/browse/DRILL-1160
> can not create hdfs as connection type in storage engine : server throws http 500 error
> ---------------------------------------------------------------------------------------
>
> Key: DRILL-1075
> URL: https://issues.apache.org/jira/browse/DRILL-1075
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Vivian Summers
> Assignee: Sudheesh Katkam
> Priority: Critical
>
> Server at 8047 throws:
> HTTP ERROR 500
> Problem accessing /storage/config/update. Reason:
> Request failed.
> configure file:
> {
> "type" : "file",
> "enabled" : true,
> "connection" : "hdfs:///",
> "workspaces" : {
> "root" : {
> "location" : "/",
> "writable" : false,
> "storageformat" : null
> },
> "default" : {
> "location" : "/user/root",
> "writable" : true,
> "storageformat" : null
> },
> "tmp" : {
> "location" : "/tmp",
> "writable" : true,
> "storageformat" : "csv"
> }
> },
> "formats" : {
> "psv" : {
> "type" : "text",
> "extensions" : [ "tbl" ],
> "delimiter" : "|"
> },
> "csv" : {
> "type" : "text",
> "extensions" : [ "csv" ],
> "delimiter" : ","
> },
> "tsv" : {
> "type" : "text",
> "extensions" : [ "tsv" ],
> "delimiter" : "\t"
> },
> "parquet" : {
> "type" : "parquet"
> },
> "json" : {
> "type" : "json"
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v6.2#6252)