You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Kevin Verhoeven <Ke...@ds-iq.com> on 2016/07/29 23:07:17 UTC

Connecting Drill to Azure Data Lake

Hi Drill Community,

Has anyone attempted to connect Drill to the Azure Data Lake? Microsoft has implemented a WebHDFS API over Azure Data Lake, so Drill should be able to connect. I'm guessing this will be similar to s3. My initial attempts have failed, does anyone have any ideas or experience with this connection?

Regards,

Kevin

RE: Connecting Drill to Azure Data Lake

Posted by Kevin Verhoeven <Ke...@ds-iq.com>.
I see the following error (Drill 1.5.0):

org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: SocketTimeoutException: connect timed out

However, I am able to connect to the Azure Data Lake from the server that reported the error using curl, so it does not appear to be a standard connectivity issue. 

My Storage configuration is very basic:

{
  "type": "file",
  "enabled": true,
  "connection": "swebhdfs://azuredatalakestoragename.azuredatalakestore.net",
  "workspaces": {
    "root": {
      "location": "/",
      "writable": true,
      "defaultInputFormat": null
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl",
        "psv"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "txt": {
      "type": "text",
      "extensions": [
        "txt"
      ],
      "delimiter": ","
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json"
    },
    "avro": {
      "type": "avro"
    }
  }
}

-----Original Message-----
From: Sudheesh Katkam [mailto:skatkam@maprtech.com] 
Sent: Monday, August 1, 2016 11:03 AM
To: user@drill.apache.org
Subject: Re: Connecting Drill to Azure Data Lake

What failure(s) do you see?

Thank you,
Sudheesh

> On Jul 29, 2016, at 4:07 PM, Kevin Verhoeven <Ke...@ds-iq.com> wrote:
> 
> Hi Drill Community,
> 
> Has anyone attempted to connect Drill to the Azure Data Lake? Microsoft has implemented a WebHDFS API over Azure Data Lake, so Drill should be able to connect. I'm guessing this will be similar to s3. My initial attempts have failed, does anyone have any ideas or experience with this connection?
> 
> Regards,
> 
> Kevin
>

Re: Connecting Drill to Azure Data Lake

Posted by Sudheesh Katkam <sk...@maprtech.com>.
What failure(s) do you see?

Thank you,
Sudheesh

> On Jul 29, 2016, at 4:07 PM, Kevin Verhoeven <Ke...@ds-iq.com> wrote:
> 
> Hi Drill Community,
> 
> Has anyone attempted to connect Drill to the Azure Data Lake? Microsoft has implemented a WebHDFS API over Azure Data Lake, so Drill should be able to connect. I'm guessing this will be similar to s3. My initial attempts have failed, does anyone have any ideas or experience with this connection?
> 
> Regards,
> 
> Kevin
>