Apache Drill + Google Storage via GCS Connector and Dataproc

[Joe Auty <jo...@thinkdataworks.com>]

Hello,

We are trying to get this combo working via the instructions found here:

https://stackoverflow.com/questions/32883965/apache-drill-using-google-cloud-storage

We are attempting this with a manually compiled version of 1.14, and can 
query GS via gsutil/hadoop commands which suggests the GCS connector is 
installed correctly and working as expected.


I don't know if there will be any takers here to help us get into the 
weeds with this particular problem (we are seeing 
"org.apache.drill.common.exceptions.UserRemoteException: VALIDATION 
ERROR: Failed to create DrillFileSystem for proxy user: No FileSystem 
for scheme: gs SQL Query null"), and of course I'm happy to provide more 
details, but if not it would also be helpful to address some more high 
level questions about this:


1) is there a way to get this to work currently?
2) if so, is there a reason why this isn't documented (e.g. is it not 
properly tested/recommended?)
3) will GS be an officially supported plugin at some point in the future?


Thanks in advance!

Re: Apache Drill + Google Storage via GCS Connector and Dataproc

[Saurabh Mahapatra <sa...@gmail.com>]

Hey Joe,

There are two options I see if you are looking for a quick turnaround on this. 

1. Use affordable consulting services to get this work done and this would include support beyond your current problems. This would include making sure that the production system works as intended and you can decide to have this contributed back to the community. Please ping me if that is something you would like to explore. I am helping out a company on similar problems with developing plugins.

2. Have you brought this up with the Google folks? I am sure they will suggest using BigQuery but you could ask them. I have some friends in the GCP platform team but nothing like having a customer like you request it through the business side. 

Best wishes,
Saurabh 

Sent from my iPhone



> On May 7, 2018, at 8:51 AM, Joe Auty <jo...@thinkdataworks.com> wrote:
> 
> Hello,
> 
> We are trying to get this combo working via the instructions found here:
> 
> https://stackoverflow.com/questions/32883965/apache-drill-using-google-cloud-storage
> 
> We are attempting this with a manually compiled version of 1.14, and can query GS via gsutil/hadoop commands which suggests the GCS connector is installed correctly and working as expected.
> 
> 
> I don't know if there will be any takers here to help us get into the weeds with this particular problem (we are seeing "org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: Failed to create DrillFileSystem for proxy user: No FileSystem for scheme: gs SQL Query null"), and of course I'm happy to provide more details, but if not it would also be helpful to address some more high level questions about this:
> 
> 
> 1) is there a way to get this to work currently?
> 2) if so, is there a reason why this isn't documented (e.g. is it not properly tested/recommended?)
> 3) will GS be an officially supported plugin at some point in the future?
> 
> 
> Thanks in advance!

Re: Apache Drill + Google Storage via GCS Connector and Dataproc

[Parth Chandra <pa...@apache.org>]

Drill uses HDFS to connect to cloud storage, so if you can connect to the
storage via hdfs, you should be able to connect using Drill.
When you configure the FileSystem storage plugin, you might need to specify
the url as "gs://" instead of "hdfs://" [1]
If you have already done so, it is possible that the cloud storage
connector us not in the Drill class path.

HTH.

[1] https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage



On Mon, May 7, 2018 at 8:51 AM, Joe Auty <jo...@thinkdataworks.com> wrote:

> Hello,
>
> We are trying to get this combo working via the instructions found here:
>
> https://stackoverflow.com/questions/32883965/apache-drill-
> using-google-cloud-storage
>
> We are attempting this with a manually compiled version of 1.14, and can
> query GS via gsutil/hadoop commands which suggests the GCS connector is
> installed correctly and working as expected.
>
>
> I don't know if there will be any takers here to help us get into the
> weeds with this particular problem (we are seeing
> "org.apache.drill.common.exceptions.UserRemoteException: VALIDATION
> ERROR: Failed to create DrillFileSystem for proxy user: No FileSystem for
> scheme: gs SQL Query null"), and of course I'm happy to provide more
> details, but if not it would also be helpful to address some more high
> level questions about this:
>
>
> 1) is there a way to get this to work currently?
> 2) if so, is there a reason why this isn't documented (e.g. is it not
> properly tested/recommended?)
> 3) will GS be an officially supported plugin at some point in the future?
>
>
> Thanks in advance!
>