You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Ron Cecchini <ro...@comcast.net> on 2020/01/29 05:41:21 UTC
Problem running Drill in a Docker container in OpenShift
Hi, all. Drill and OpenShift newbie here.
Has anyone successfully deployed a Drill Docker container to an OpenShift environment?
While there is information about Drill Docker, there seems to be zero information about OpenShift in particular.
Per the instructions at drill.apache.org/docs/running-drill-on-docker, I pulled the Drill Docker image from Docker Hub, and then pushed it to our OpenShift environment. But when I tried to deploy it, I immediately ran into an error about /opt/drill/conf/drill-override.conf not being readable.
I understand why the problem is happening (because of who OpenShift runs the container as), so I downloaded the source from GitHub and modified the Dockerfile to include:
RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
so that all of /opt/drill would be available to everyone. But then 'docker build' kept failing, giving the error:
Non-resolvable parent POM for org.apache.drill:drill-root:1.18.0-SNAPSHOT:
Could not transfer artifact org.apache:apache:pom:21
I tried researching that error but couldn't figure out what was going on. So I finally decided to start trying to mount persistent volumes, creating one PV for /opt/drill/conf (and then copying the default drill-override.conf there) and one PV for /opt/drill/log.
Now the container gets much further, but eventually fails on something Hadoop related. I'm not trying to do anything with Hadoop, so I don't know what that's about, but it says I don't have HADOOP_HOME set.
Hopefully I can figure out the remaining steps I need (an environment variable? more configs?), but I was wondering if anyone else had already successfully figured out how to deploy to OpenShift, or might know why the 'docker build' fails with that error?
For what it's worth, I copied over only that drill-override.conf and nothing else. And I did not set any Drill environment variables in OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
Thanks for any help!
Ron
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
Thanks, Paul.
You're right. If I had thought about it a little more and put 2 & 2 together, I would have realized that we don't even need SqlLine up and running, as our use case is simply:
Get a single Drill server running, and have a single app connect to it via JDBC calls.
(This app in our processing pipeline is responsible for both persisting data and fetching data to support a UI.)
The analyst(s) who want to play with the data already have a bunch of Python scripts to fetch data (from Drill, Mongo, etc) to do their stuff.
So as you were suggesting, the real issue boils down to figuring out how to get just the server containerized -- which I see you and others are already tackling.
Currently, we're going ahead with an uncontainerized Drill server (and I have to get back to finishing this JDBC code to make Drill calls...), so I won't bother you with any more questions. But I definitely am interested in working with you and testing out any container solutions you come up with (ultimately the powers that be want everything containerized as much as possible).
Thank you again.
Ron
> On February 1, 2020 at 2:05 AM Paul Rogers <pa...@yahoo.com.INVALID> wrote:
>
>
> Hi Ron,
>
> I suspect you are right about the issue being TTY related. Sounds like you are bringing up SqlLine inside the pod. But, Sqlline is a command line app that runs Drill in embedded mode. I wonder what Sqline is using as stdin?
>
> I personally have never used Docker to bring up clients; only servers. So, if we can get the right Dockerfile setup, you might be better off running a Drill server inside the pod, then run Sqlline from your laptop or some terminal session, and have it connect to the Drill server. You can even use K8s to ssh into the pod and start Sqlline there. If your shell exists, Sqlline will exit, but the Drill server will continue to run.
>
> Running the Drill server (Drillbit) in a pod might be a good idea in general. Once you have your app up and running, you will want multiple people to connect to your running Drill server using Sqlline (which uses JDBC), ODBC, JDBC directly, etc. Also, once you have the server up, you can use a K8s proxy (or more ingress sophisticated mechanism, you have one) to connect to the Drill web console to run queries, look at query profiles, monitor Drill status, etc. (K8s runs its own network layer so, for "stock" K8s, pod IPs are not visible outside of K8s. Maybe OpenShift has an ingress solution built in.)
>
> Thanks,
> - Paul
>
>
>
> On Friday, January 31, 2020, 9:21:49 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
>
> FWIW, I found a thread on SO that addresses this OpenShift issue with containers seemingly exiting for no apparent reason, even when running as root. It might be TTY related.
>
> openshift pod fails and restarts frequently
> https://stackoverflow.com/questions/35744404/openshift-pod-fails-and-restarts-frequently/36234707#36234707
>
> What is CrashLoopBackOff status for openshift pods?
> https://stackoverflow.com/questions/35710965/what-is-crashloopbackoff-status-for-openshift-pods
>
> > On January 31, 2020 at 4:40 PM Ron Cecchini <ro...@comcast.net> wrote:
> >
> >
> > Thank you, Paul, for your in depth and informative response.
> >
> > Here's what we did today as a test:
> >
> > In OpenShift, we enabled allowing containers to run however they wanted and redeployed the Drill Docker.
>
> [...]
>
> > The result is that the container comes up - shows that the user.name is indeed root - doesn't give any errors ...
> >
> > And then seemingly goes away, again w/o any indication of an error. And then OpenShift goes into its crash-reboot cycle trying to restart the container, and each time we see the same thing:
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Ron,
I suspect you are right about the issue being TTY related. Sounds like you are bringing up SqlLine inside the pod. But, Sqlline is a command line app that runs Drill in embedded mode. I wonder what Sqline is using as stdin?
I personally have never used Docker to bring up clients; only servers. So, if we can get the right Dockerfile setup, you might be better off running a Drill server inside the pod, then run Sqlline from your laptop or some terminal session, and have it connect to the Drill server. You can even use K8s to ssh into the pod and start Sqlline there. If your shell exists, Sqlline will exit, but the Drill server will continue to run.
Running the Drill server (Drillbit) in a pod might be a good idea in general. Once you have your app up and running, you will want multiple people to connect to your running Drill server using Sqlline (which uses JDBC), ODBC, JDBC directly, etc. Also, once you have the server up, you can use a K8s proxy (or more ingress sophisticated mechanism, you have one) to connect to the Drill web console to run queries, look at query profiles, monitor Drill status, etc. (K8s runs its own network layer so, for "stock" K8s, pod IPs are not visible outside of K8s. Maybe OpenShift has an ingress solution built in.)
Thanks,
- Paul
On Friday, January 31, 2020, 9:21:49 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
FWIW, I found a thread on SO that addresses this OpenShift issue with containers seemingly exiting for no apparent reason, even when running as root. It might be TTY related.
openshift pod fails and restarts frequently
https://stackoverflow.com/questions/35744404/openshift-pod-fails-and-restarts-frequently/36234707#36234707
What is CrashLoopBackOff status for openshift pods?
https://stackoverflow.com/questions/35710965/what-is-crashloopbackoff-status-for-openshift-pods
> On January 31, 2020 at 4:40 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> Thank you, Paul, for your in depth and informative response.
>
> Here's what we did today as a test:
>
> In OpenShift, we enabled allowing containers to run however they wanted and redeployed the Drill Docker.
[...]
> The result is that the container comes up - shows that the user.name is indeed root - doesn't give any errors ...
>
> And then seemingly goes away, again w/o any indication of an error. And then OpenShift goes into its crash-reboot cycle trying to restart the container, and each time we see the same thing:
Re: Problem running Drill in a Docker container in OpenShift
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi All,
Our current embedded Docker container works best when run from Docker with the "-it" (interactive) flag. Since K8s pods are seldom interactive, it seems we'll need to create a container focused on the needs of server deployment. DRILL-7563 [1] has an outline of the tasks. Fortunately, the steps are pretty standard; many projects have led the way.
A first-draft Drillbit container is hosted at [2], source code is at [3]. (These are personal repos for now.) The basics work under Docker: I can connect to the Drill Web Console and run a query. The key dependency is ZK [4]. The next step is to create a sqlline container that can be run in interactive mode and which connects to the containerized Drillbit(s).
K8s has nice ways to pass configuration into a container, which we'll need to run the Drillbit with other than the out-of-the-box configuration. (Memory will be the first thing someone wants to change.) Rather than fight with the clunky Docker solutions to config, I'll just jump directly over to the K8s approach.
Suggestions, feedback and help welcome.
Thanks,
- Paul
[1] https://issues.apache.org/jira/browse/DRILL-7563
[2] https://hub.docker.com/repository/docker/gaucho84/drill
[3] https://github.com/paul-rogers/drill-docker See the docker folder for details.
[4] https://hub.docker.com/_/zookeeper/
On Friday, January 31, 2020, 9:21:49 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
FWIW, I found a thread on SO that addresses this OpenShift issue with containers seemingly exiting for no apparent reason, even when running as root. It might be TTY related.
openshift pod fails and restarts frequently
https://stackoverflow.com/questions/35744404/openshift-pod-fails-and-restarts-frequently/36234707#36234707
What is CrashLoopBackOff status for openshift pods?
https://stackoverflow.com/questions/35710965/what-is-crashloopbackoff-status-for-openshift-pods
> On January 31, 2020 at 4:40 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> Thank you, Paul, for your in depth and informative response.
>
> Here's what we did today as a test:
>
> In OpenShift, we enabled allowing containers to run however they wanted and redeployed the Drill Docker.
[...]
> The result is that the container comes up - shows that the user.name is indeed root - doesn't give any errors ...
>
> And then seemingly goes away, again w/o any indication of an error. And then OpenShift goes into its crash-reboot cycle trying to restart the container, and each time we see the same thing:
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
FWIW, I found a thread on SO that addresses this OpenShift issue with containers seemingly exiting for no apparent reason, even when running as root. It might be TTY related.
openshift pod fails and restarts frequently
https://stackoverflow.com/questions/35744404/openshift-pod-fails-and-restarts-frequently/36234707#36234707
What is CrashLoopBackOff status for openshift pods?
https://stackoverflow.com/questions/35710965/what-is-crashloopbackoff-status-for-openshift-pods
> On January 31, 2020 at 4:40 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> Thank you, Paul, for your in depth and informative response.
>
> Here's what we did today as a test:
>
> In OpenShift, we enabled allowing containers to run however they wanted and redeployed the Drill Docker.
[...]
> The result is that the container comes up - shows that the user.name is indeed root - doesn't give any errors ...
>
> And then seemingly goes away, again w/o any indication of an error. And then OpenShift goes into its crash-reboot cycle trying to restart the container, and each time we see the same thing:
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
Thank you, Paul, for your in depth and informative response.
Here's what we did today as a test:
In OpenShift, we enabled allowing containers to run however they wanted and redeployed the Drill Docker.
Note:
- this is the 1.17.0 image we pulled directly from Docker Hub, not the image I made from that image
- we are using 2 persistent volumes (PV) for /opt/drill/conf and /opt/drill/log
- I copied everything that was in the source code's (from GitHub) 'resources' directory into the PV mapped to /opt/drill/conf. (FWIW, there are more files in that 'resources' directory than in the container's original /opt/drill/conf.)
The result is that the container comes up - shows that the user.name is indeed root - doesn't give any errors ...
And then seemingly goes away, again w/o any indication of an error. And then OpenShift goes into its crash-reboot cycle trying to restart the container, and each time we see the same thing:
The console in OpenShift displays:
Apache Drill 1.17.0
"You won't meet Santa, but Drill supports Clauses."
which obviously looks good.
But the sqlline.log, while not indicating anything bad, seems to go into a graceful shutdown immediately after connecting. I turned up logging on sqlline to DEBUG and will include the tail end of the log below.
The only other clue I have is that if I go into the OpenShift debug shell for the Drill container, and execute /opt/drill/bin/drill-embedded, then Drill comes up and stays up (only while that debug shell is up).
In the meantime, I think we may be looking to install Drill from the tar.gz and run it on a dedicated server outside of OpenShift.
Thank you again.
-----
[...]
2020-01-31 21:32:54,030 [main] DEBUG o.a.d.e.w.f.r.ResourceManagerBuilder - No query queueing enabled.
2020-01-31 21:32:54,032 [main] INFO o.apache.drill.exec.server.Drillbit - Startup completed (5787 ms).
2020-01-31 21:32:54,034 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.used
2020-01-31 21:32:54,034 [main] WARN o.a.drill.exec.metrics.DrillMetrics - Removing old metric since name matched newly registered metric. Metric name: drill.allocator.root.peak
2020-01-31 21:32:54,048 [main] DEBUG org.apache.drill.exec.ssl.SSLConfig - Initialized SSL context.
2020-01-31 21:32:54,048 [main] DEBUG o.a.drill.exec.client.DrillClient - Connecting to server XXXXXXX-16-6xqlt:31010
2020-01-31 21:32:54,140 [UserServer-1] DEBUG o.a.drill.exec.memory.BoundsChecking - Direct memory bounds checking is disabled.
2020-01-31 21:32:54,152 [Client-1] DEBUG o.a.d.e.rpc.ConnectionMultiListener - Handshake completed successfully.
2020-01-31 21:32:54,153 [main] INFO o.a.drill.exec.client.DrillClient - Foreman drillbit is XXXXXXX-16-6xqlt
2020-01-31 21:32:54,153 [main] INFO o.a.drill.exec.client.DrillClient - Successfully connected to server XXXXXXX-16-6xqlt:31010
2020-01-31 21:32:54,382 [main] DEBUG o.apache.drill.exec.rpc.BasicClient - Closing client
2020-01-31 21:32:54,385 [main] DEBUG o.apache.drill.exec.server.Drillbit - Shutdown begun.
2020-01-31 21:32:55,416 [main] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 0, cache miss count: 0, hit rate: 0%
2020-01-31 21:32:55,428 [main] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1042 ms).
2020-01-31 21:32:55,432 [Drillbit-Graceful-Shutdown#Thread-6] DEBUG o.apache.drill.exec.server.Drillbit - Graceful Shutdown thread was interrupted
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118)
at org.apache.drill.exec.server.Drillbit$GracefulShutdownThread.pollShutdown(Drillbit.java:427)
at org.apache.drill.exec.server.Drillbit$GracefulShutdownThread.run(Drillbit.java:401)
> On January 31, 2020 at 2:06 AM Paul Rogers <pa...@yahoo.com.INVALID> wrote:
>
>
> Hi Ron,
>
> Actually, helping us track down this issue is a great contribution in itself.
>
> My Docker is a bit rusty, but I just did a quick check of Drill's two Dockerfiles. Neither seem designed for a production deployment: one is a build image, the other a simple embedded image. Neither set a user.
>
> I suspect we need to create an image for this use case that:
>
> 1. Installs dependencies (JDK, etc.)
> 2. Creates a "drill" user.
> 3. Downloads and unpacks the target Drill distribution.
> 4. Changes owner of the distribution to user "drill".
> 5. Runs Drill as "user drill."
>
> For historical reasons, lots of Drill QA and developers deployed Drill as root when running on VMs, looks like the Docker image may have followed that pattern. But, as you noted, running anything as root, even in a container, is usually frowned upon.
>
> I did a quick check of some other community Dockerfiles. As expected, they mostly follow the same steps as above (customized, of course, for the particular project.)
>
>
> I think you said you are new to Docker. You can try to create an image that does the above; it's not hard (just lots of details to learn). Otherwise, I may have some time to take a crack at it in a few days. Or, perhaps one of the other Drill devs. can try it sooner.
>
>
> Your goal is to run Drill as part of your app. To do that, we'll want to option to host-mount certain directories. For example, it might be helpful to write query profiles and logs to a host directory so that these files survive a container failure.
>
> OpenShift is a wrapper around Kubernetes. So, our Drill image should be designed to run under K8s. For example, we should use K8s config maps to pass in things like the Drill config file (so you can run Drill distributed, pointed to a ZK), and to use K8s to set things like the Drill memory options. (Otherwise, you'd have to rebuild the container to grab a new drill-override.conf or drill-overide.sh file any time you want to change a setting. OK for playing around, not scalable in production.
>
> Your data will be stored somewhere. Ceph? Using its S3 look-alike API? If so, you'll need to pass that config into your container and define the required storage plugin. You can do that statically each time, or you can run ZK so that storage plugins and other config survive from one container run to the next.
>
>
> I can take a crack at adding the needed Dockerfile features, and the K8s setup, once we get the basics to work. Or, if another Drill dev wants to try, start with a one-node K8s on your PC or test machine.
>
> In the mean time, one of the nice things about Drill is you can actually try things out on your own laptop or VM. Grab a few Parquet files of the type you plan to use. Or, configure drill to read from your distributed file system. Run some test queries. Get the hang of how Drill works so you are comfortable with using Drill once we get your containers to run.
>
>
> Thanks,
> - Paul
>
>
>
> On Thursday, January 30, 2020, 10:14:08 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
>
> So, apparently I spoke too soon, as my image-from-an-image in OpenShift actually *didn't* start up successfully.
>
> I'll take Ted's and Charles' comments in the current "[DISCUSS]" thread regarding attracting users who may never contribute a line of code but nevertheless report on their use of Drill and problems they may encounter as license to ask a follow up question... Besides, with the growing usage of OpenShift, I'm sure I won't be the last one trying to do this!
>
> You guys were talking about attracting enthusiastic users who want to spread the word to their friends. Well, as a bit of a backstory, it was an enthusiastic analyst here who floated the idea of replacing our MongoDB + JSON with Drill + Parquet. Eventually our tech lead signed on to the idea and I got tasked to help out. If this all works out, I'm sure word will spread to other analysts. So here I am... (I don't mind being the "canary in the coalmine", so to speak. I've often had to take on that role. And I get to learn new things...)
>
> Ok, enough blah-blah-blah. The problem I'm having, as far as I can tell, stems from the fact that OpenShift doesn't set a user name when running a container. In particular, the Java System.getProperty("user.name") and "user.home" calls return "?" as reported in the Zookeeper (I'm in embedded mode but the Zookeeper config is still being filled in) section of the sqlline.log:
>
> [main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=?
> [main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=?
>
> As a quick point of comparison, when I run a Drill Docker on my desktop (not in OpenShift), with zero config changes, everything of course works fine, and "user.name" is "root" and "user.home" is "/root". (Probably because I installed Docker with a "sudo yum install".) Similarly, when I run an embedded-Drill on my desktop installed from the source *tar.gz - i.e. not in a Docker, and not in OpenShift - "user.name" is my $USER and "user.home" is my $HOME, and everything again runs fine.
>
> I'll include the big stack trace at the bottom, but really the only question I have at the moment is:
>
> Given that I don't think I can get OpenShift to set or pass in a "user.name" property for the JDK to get, is there an environment variable (OpenShift *can* pass those in) or a magical setting in drill-override.conf that will let me get past this "invalid null input: name" problem I'm running into during the "login" phase of the startup?
>
> And if so, could the "user.name" be set to anything or would it have to be root? (I don't quite understand the relationship between the "user.name" and the Hadoop login, etc.)
>
> Thank you so much for any help!
>
> Ron
>
> --------------------------------------------------------------------------------
>
> Error: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login. (state=,code=0)
> java.sql.SQLException: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:143)
> at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
> at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
> at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
> at sqlline.Commands.connect(Commands.java:1364)
> at sqlline.Commands.connect(Commands.java:1244)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:730)
> at sqlline.SqlLine.initArgs(SqlLine.java:410)
> at sqlline.SqlLine.begin(SqlLine.java:515)
> at sqlline.SqlLine.start(SqlLine.java:267)
> at sqlline.SqlLine.main(SqlLine.java:206)
> Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
> at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:161)
> at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
> ... 18 more
> Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
> at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
> at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
> at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
> at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
> at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
> at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
> at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
> at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
> at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
> at sqlline.Commands.connect(Commands.java:1364)
> at sqlline.Commands.connect(Commands.java:1244)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:730)
> at sqlline.SqlLine.initArgs(SqlLine.java:410)
> at sqlline.SqlLine.begin(SqlLine.java:515)
> at sqlline.SqlLine.start(SqlLine.java:267)
> at sqlline.SqlLine.main(SqlLine.java:206)
>
> at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
> at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
> at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
> ... 22 more
> Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
> at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
> at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
> at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
> at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
> at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
> at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
> at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
> at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
> at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
> at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
> at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
> at sqlline.Commands.connect(Commands.java:1364)
> at sqlline.Commands.connect(Commands.java:1244)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:730)
> at sqlline.SqlLine.initArgs(SqlLine.java:410)
> at sqlline.SqlLine.begin(SqlLine.java:515)
> at sqlline.SqlLine.start(SqlLine.java:267)
> at sqlline.SqlLine.main(SqlLine.java:206)
>
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
> at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
> ... 25 more
> Apache Drill 1.17.0
> "You told me to, Drill Sergeant!"
>
> > On January 29, 2020 at 5:47 PM Ron Cecchini <ro...@comcast.net> wrote:
> >
> > Sorry for the spam, but I think I figured it out.
> >
> > Thank you so much for your suggestions to build an image from an image. I finally put 2 & 2 together and realized what you were saying and created the following Dockerfile. I then built and pushed the image into OpenShift - and it started up nicely.
> >
> > I haven't had a chance to test it yet, but I'm optimistic.
> >
> > Thank you again.
> >
> > ---
> >
> > Dockerfile:
> >
> > # Use the latest official release of Apache Drill
> > FROM apache/drill:1.17.0
> >
> > # Make /opt/drill readable by everyone due to OpenShift's security
> > RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
> >
> > # Start Drill in embedded mode and connects to Sqlline
> > ENTRYPOINT /opt/drill/bin/drill-embedded
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Ron,
Actually, helping us track down this issue is a great contribution in itself.
My Docker is a bit rusty, but I just did a quick check of Drill's two Dockerfiles. Neither seem designed for a production deployment: one is a build image, the other a simple embedded image. Neither set a user.
I suspect we need to create an image for this use case that:
1. Installs dependencies (JDK, etc.)
2. Creates a "drill" user.
3. Downloads and unpacks the target Drill distribution.
4. Changes owner of the distribution to user "drill".
5. Runs Drill as "user drill."
For historical reasons, lots of Drill QA and developers deployed Drill as root when running on VMs, looks like the Docker image may have followed that pattern. But, as you noted, running anything as root, even in a container, is usually frowned upon.
I did a quick check of some other community Dockerfiles. As expected, they mostly follow the same steps as above (customized, of course, for the particular project.)
I think you said you are new to Docker. You can try to create an image that does the above; it's not hard (just lots of details to learn). Otherwise, I may have some time to take a crack at it in a few days. Or, perhaps one of the other Drill devs. can try it sooner.
Your goal is to run Drill as part of your app. To do that, we'll want to option to host-mount certain directories. For example, it might be helpful to write query profiles and logs to a host directory so that these files survive a container failure.
OpenShift is a wrapper around Kubernetes. So, our Drill image should be designed to run under K8s. For example, we should use K8s config maps to pass in things like the Drill config file (so you can run Drill distributed, pointed to a ZK), and to use K8s to set things like the Drill memory options. (Otherwise, you'd have to rebuild the container to grab a new drill-override.conf or drill-overide.sh file any time you want to change a setting. OK for playing around, not scalable in production.
Your data will be stored somewhere. Ceph? Using its S3 look-alike API? If so, you'll need to pass that config into your container and define the required storage plugin. You can do that statically each time, or you can run ZK so that storage plugins and other config survive from one container run to the next.
I can take a crack at adding the needed Dockerfile features, and the K8s setup, once we get the basics to work. Or, if another Drill dev wants to try, start with a one-node K8s on your PC or test machine.
In the mean time, one of the nice things about Drill is you can actually try things out on your own laptop or VM. Grab a few Parquet files of the type you plan to use. Or, configure drill to read from your distributed file system. Run some test queries. Get the hang of how Drill works so you are comfortable with using Drill once we get your containers to run.
Thanks,
- Paul
On Thursday, January 30, 2020, 10:14:08 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
So, apparently I spoke too soon, as my image-from-an-image in OpenShift actually *didn't* start up successfully.
I'll take Ted's and Charles' comments in the current "[DISCUSS]" thread regarding attracting users who may never contribute a line of code but nevertheless report on their use of Drill and problems they may encounter as license to ask a follow up question... Besides, with the growing usage of OpenShift, I'm sure I won't be the last one trying to do this!
You guys were talking about attracting enthusiastic users who want to spread the word to their friends. Well, as a bit of a backstory, it was an enthusiastic analyst here who floated the idea of replacing our MongoDB + JSON with Drill + Parquet. Eventually our tech lead signed on to the idea and I got tasked to help out. If this all works out, I'm sure word will spread to other analysts. So here I am... (I don't mind being the "canary in the coalmine", so to speak. I've often had to take on that role. And I get to learn new things...)
Ok, enough blah-blah-blah. The problem I'm having, as far as I can tell, stems from the fact that OpenShift doesn't set a user name when running a container. In particular, the Java System.getProperty("user.name") and "user.home" calls return "?" as reported in the Zookeeper (I'm in embedded mode but the Zookeeper config is still being filled in) section of the sqlline.log:
[main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=?
[main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=?
As a quick point of comparison, when I run a Drill Docker on my desktop (not in OpenShift), with zero config changes, everything of course works fine, and "user.name" is "root" and "user.home" is "/root". (Probably because I installed Docker with a "sudo yum install".) Similarly, when I run an embedded-Drill on my desktop installed from the source *tar.gz - i.e. not in a Docker, and not in OpenShift - "user.name" is my $USER and "user.home" is my $HOME, and everything again runs fine.
I'll include the big stack trace at the bottom, but really the only question I have at the moment is:
Given that I don't think I can get OpenShift to set or pass in a "user.name" property for the JDK to get, is there an environment variable (OpenShift *can* pass those in) or a magical setting in drill-override.conf that will let me get past this "invalid null input: name" problem I'm running into during the "login" phase of the startup?
And if so, could the "user.name" be set to anything or would it have to be root? (I don't quite understand the relationship between the "user.name" and the Hadoop login, etc.)
Thank you so much for any help!
Ron
--------------------------------------------------------------------------------
Error: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login. (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:143)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:161)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
... 18 more
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
... 22 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
... 25 more
Apache Drill 1.17.0
"You told me to, Drill Sergeant!"
> On January 29, 2020 at 5:47 PM Ron Cecchini <ro...@comcast.net> wrote:
>
> Sorry for the spam, but I think I figured it out.
>
> Thank you so much for your suggestions to build an image from an image. I finally put 2 & 2 together and realized what you were saying and created the following Dockerfile. I then built and pushed the image into OpenShift - and it started up nicely.
>
> I haven't had a chance to test it yet, but I'm optimistic.
>
> Thank you again.
>
> ---
>
> Dockerfile:
>
> # Use the latest official release of Apache Drill
> FROM apache/drill:1.17.0
>
> # Make /opt/drill readable by everyone due to OpenShift's security
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> # Start Drill in embedded mode and connects to Sqlline
> ENTRYPOINT /opt/drill/bin/drill-embedded
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
So, apparently I spoke too soon, as my image-from-an-image in OpenShift actually *didn't* start up successfully.
I'll take Ted's and Charles' comments in the current "[DISCUSS]" thread regarding attracting users who may never contribute a line of code but nevertheless report on their use of Drill and problems they may encounter as license to ask a follow up question... Besides, with the growing usage of OpenShift, I'm sure I won't be the last one trying to do this!
You guys were talking about attracting enthusiastic users who want to spread the word to their friends. Well, as a bit of a backstory, it was an enthusiastic analyst here who floated the idea of replacing our MongoDB + JSON with Drill + Parquet. Eventually our tech lead signed on to the idea and I got tasked to help out. If this all works out, I'm sure word will spread to other analysts. So here I am... (I don't mind being the "canary in the coalmine", so to speak. I've often had to take on that role. And I get to learn new things...)
Ok, enough blah-blah-blah. The problem I'm having, as far as I can tell, stems from the fact that OpenShift doesn't set a user name when running a container. In particular, the Java System.getProperty("user.name") and "user.home" calls return "?" as reported in the Zookeeper (I'm in embedded mode but the Zookeeper config is still being filled in) section of the sqlline.log:
[main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=?
[main] INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=?
As a quick point of comparison, when I run a Drill Docker on my desktop (not in OpenShift), with zero config changes, everything of course works fine, and "user.name" is "root" and "user.home" is "/root". (Probably because I installed Docker with a "sudo yum install".) Similarly, when I run an embedded-Drill on my desktop installed from the source *tar.gz - i.e. not in a Docker, and not in OpenShift - "user.name" is my $USER and "user.home" is my $HOME, and everything again runs fine.
I'll include the big stack trace at the bottom, but really the only question I have at the moment is:
Given that I don't think I can get OpenShift to set or pass in a "user.name" property for the JDK to get, is there an environment variable (OpenShift *can* pass those in) or a magical setting in drill-override.conf that will let me get past this "invalid null input: name" problem I'm running into during the "login" phase of the startup?
And if so, could the "user.name" be set to anything or would it have to be root? (I don't quite understand the relationship between the "user.name" and the Hadoop login, etc.)
Thank you so much for any help!
Ron
--------------------------------------------------------------------------------
Error: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login. (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:143)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:161)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
... 18 more
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
... 22 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
at sqlline.Commands.connect(Commands.java:1364)
at sqlline.Commands.connect(Commands.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:730)
at sqlline.SqlLine.initArgs(SqlLine.java:410)
at sqlline.SqlLine.begin(SqlLine.java:515)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
... 25 more
Apache Drill 1.17.0
"You told me to, Drill Sergeant!"
> On January 29, 2020 at 5:47 PM Ron Cecchini <ro...@comcast.net> wrote:
>
> Sorry for the spam, but I think I figured it out.
>
> Thank you so much for your suggestions to build an image from an image. I finally put 2 & 2 together and realized what you were saying and created the following Dockerfile. I then built and pushed the image into OpenShift - and it started up nicely.
>
> I haven't had a chance to test it yet, but I'm optimistic.
>
> Thank you again.
>
> ---
>
> Dockerfile:
>
> # Use the latest official release of Apache Drill
> FROM apache/drill:1.17.0
>
> # Make /opt/drill readable by everyone due to OpenShift's security
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> # Start Drill in embedded mode and connects to Sqlline
> ENTRYPOINT /opt/drill/bin/drill-embedded
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
Sorry for the spam, but I think I figured it out.
Thank you so much for your suggestions to build an image from an image. I finally put 2 & 2 together and realized what you were saying and created the following Dockerfile. I then built and pushed the image into OpenShift - and it started up nicely.
I haven't had a chance to test it yet, but I'm optimistic.
Thank you again.
---
Dockerfile:
# Use the latest official release of Apache Drill
FROM apache/drill:1.17.0
# Make /opt/drill readable by everyone due to OpenShift's security
RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
# Start Drill in embedded mode and connects to Sqlline
ENTRYPOINT /opt/drill/bin/drill-embedded
> On January 29, 2020 at 3:05 PM Ron Cecchini <ro...@comcast.net> wrote:
>
>
> Thank you, Paul and Volodymyr. I'll answer all of your questions below. (Warning: It gets a little long...)
>
> So, first of all, I am behind a proxied firewall, and I am trying to do this build on my RHEL/CentOS 7 development machine, and then replicate the steps on another ("build") machine and deploy it into an OpenShift environment, both of which are also running on CentOS 7.
>
> And we are looking to run Drill only in embedded mode.
>
> > Please explain a bit more about the file permissions issue.
> > Is the file owned by a user other than the one that runs Drill?
> > If so, sounds like a bug, unless OpenShift uses a different user than plain Docker would.
>
> Per the OpenShift docs below, OpenShift indeed runs the container as a different user than Docker would.
>
> https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines
>
> Support Arbitrary User IDs
>
> By default, OpenShift Container Platform runs containers using an arbitrarily assigned user ID.
> This provides additional security against processes escaping the container due to a container
> engine vulnerability and thereby achieving escalated permissions on the host node.
>
> For an image to support running as an arbitrary user, directories and files that may be written
> to by processes in the image should be owned by the root group and be read/writable by that group.
> Files to be executed should also have group execute permissions.
>
> Adding the following to your Dockerfile sets the directory and file permissions to allow users
> in the root group to access them in the built image:
>
> RUN chgrp -R 0 /some/directory && chmod -R g=u /some/directory
>
> ...
>
> OpenShift does provide a way to run containers as root -- but we're absolutely trying to avoid that.
>
> So it was that suggestion to add the 'chgrp' and 'chmod' to the Dockerfile that seemed to be the best and easiest solution.
>
> > I believe our standard image is for building Drill.
> > What you want is an image that uses an existing Drill build.
> > The "SNAPSHOT" refers to the current master version of the code, which is probably not what you want.
> > You want the released 1.17 binaries since you want to use, not develop, Drill.
>
> Thank you for the suggestion. You were right, I was sitting on the master branch.
>
> So I checked out the 1.17.0 branch.
>
> Then I saw that there was no Dockerfile in this branch, so I copied the one I had been using (including the added chgrp/chmod), tried the 'docker build' -- and again got a similar build error regarding the parent pom.
>
> But looking at the error, it seems to be more an issue with finding the specified Apache POM in the conjars.org/repo. The following simply does not exist:
>
> http://conjars.org/repo/org/apache/apache/21/apache-21.pom
>
> (And on a related sidenote, I've built the apache/camel project, and its parent POM has an essentially identical '<parent><groupId>org.apache' section as yours - and it doesn't barf with a "parent POM" problem - so that can't be the problem. It's obviously something to do with this connection to conjars.org.)
>
> This is how I tried building the Drill Docker and the FATAL error I get:
>
> % git branch
>
> * (detached from origin/1.17.0)
> master
>
> % docker build -t mydrill .
>
> Sending build context to Docker daemon 199.2MB
> Step 1/10 : FROM maven:3.6-jdk-8 as build
> ---> 5042e76d4104
> Step 2/10 : COPY . /src
> ---> 9a373ca8c131
> Step 3/10 : WORKDIR /src
> ---> Running in 982bd7f24911
> Removing intermediate container 982bd7f24911
> ---> 0b023e6084b1
> Step 4/10 : RUN mvn clean install -DskipTests -q
> ---> Running in f2840c1bb274
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.17.0:
> Could not transfer artifact org.apache:apache:pom:21 from/to conjars (http://conjars.org/repo):
> Transfer failed for http://conjars.org/repo/org/apache/apache/21/apache-21.pom and
> 'parent.relativePath' points at no local POM @ line 24, column 11:
> Connect to conjars.org:80 [conjars.org/184.73.255.175, conjars.org/54.225.137.155] failed:
> Connection timed out (Connection timed out) -> [Help 2]
>
>
> > Short term, the best solution would be if you can build an image based on the existing Drill image rather than making a copy of the Dockerfile.
>
> Since I'm essentially new to Docker as well, how would I build a new image -- using the 1.17.0 image I already have in my local Docker repo (instead of building from source) -- and also doing the chgrp/chmod I need to do?
>
> (I did figure out how to use 'docker save apache/drill > apache-drill.tar' to pull apart a container, and inspected all the layers to find the apache-drill-1.17.0.tar.gz you're referring to. But hopefully I don't need to go down *that* path of trying to build an image from scratch using your jars, and the 3rd party jars you include, etc, and hopefully there's a simpler way of building an image based on your image.)
>
> Or maybe the solution to the conjars.org maven build error is trivial and I can try that again.
>
> Thank you again so much!
>
> Ron
>
> > If we understand the original file permission problem, perhaps we can find a way to fix that.
>
> > On January 29, 2020 at 2:10 AM Paul Rogers <pa...@yahoo.com.INVALID> wrote:
> >
> >
> > Hi Ron,
> >
> > I don't think anyone on the Drill team has access to an OpenShift environment. Let's see if we can use your work to ensure that the Docker image supports OpenShift in the future.
> >
> > Please explain a bit more about the file permissions issue. Is the file owned by a user other than the one that runs Drill? If so, sounds like a bug, unless OpenShift uses a different user than plain Docker would.
> >
> >
> > I believe our standard image is for building Drill. What you want is an image that uses an existing Drill build. The "SNAPSHOT" refers to the current master version of the code, which is probably not what you want. You want the released 1.17 binaries since you want to use, not develop, Drill.
> >
> > Question for the team: do we have a separate image for folks who want to run the latest 1.17 release?
> >
> > Short term, the best solution would be if you can build an image based on the existing Drill image rather than making a copy of the Dockerfile. If we understand the original file permission problem, perhaps we can find a way to fix that.
> >
> > Are you looking to run Drill in embedded mode (Sqlline in a container, you ssh into the container; config lost on Drill shutdown) or in server mode (config stored in ZK so it persists across container runs)?
> >
> >
> > Thanks,
> > - Paul
> >
> >
> >
> > On Tuesday, January 28, 2020, 9:41:28 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
> >
> >
> > Hi, all. Drill and OpenShift newbie here.
> >
> > Has anyone successfully deployed a Drill Docker container to an OpenShift environment?
> >
> > While there is information about Drill Docker, there seems to be zero information about OpenShift in particular.
> >
> > Per the instructions at drill.apache.org/docs/running-drill-on-docker, I pulled the Drill Docker image from Docker Hub, and then pushed it to our OpenShift environment. But when I tried to deploy it, I immediately ran into an error about /opt/drill/conf/drill-override.conf not being readable.
> >
> > I understand why the problem is happening (because of who OpenShift runs the container as), so I downloaded the source from GitHub and modified the Dockerfile to include:
> >
> > RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
> >
> > so that all of /opt/drill would be available to everyone. But then 'docker build' kept failing, giving the error:
> >
> > Non-resolvable parent POM for org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> > Could not transfer artifact org.apache:apache:pom:21
> >
> > I tried researching that error but couldn't figure out what was going on. So I finally decided to start trying to mount persistent volumes, creating one PV for /opt/drill/conf (and then copying the default drill-override.conf there) and one PV for /opt/drill/log.
> >
> > Now the container gets much further, but eventually fails on something Hadoop related. I'm not trying to do anything with Hadoop, so I don't know what that's about, but it says I don't have HADOOP_HOME set.
> >
> > Hopefully I can figure out the remaining steps I need (an environment variable? more configs?), but I was wondering if anyone else had already successfully figured out how to deploy to OpenShift, or might know why the 'docker build' fails with that error?
> >
> > For what it's worth, I copied over only that drill-override.conf and nothing else. And I did not set any Drill environment variables in OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
> >
> > Thanks for any help!
> >
> > Ron
> >
Re: Problem running Drill in a Docker container in OpenShift
Posted by Ron Cecchini <ro...@comcast.net>.
Thank you, Paul and Volodymyr. I'll answer all of your questions below. (Warning: It gets a little long...)
So, first of all, I am behind a proxied firewall, and I am trying to do this build on my RHEL/CentOS 7 development machine, and then replicate the steps on another ("build") machine and deploy it into an OpenShift environment, both of which are also running on CentOS 7.
And we are looking to run Drill only in embedded mode.
> Please explain a bit more about the file permissions issue.
> Is the file owned by a user other than the one that runs Drill?
> If so, sounds like a bug, unless OpenShift uses a different user than plain Docker would.
Per the OpenShift docs below, OpenShift indeed runs the container as a different user than Docker would.
https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines
Support Arbitrary User IDs
By default, OpenShift Container Platform runs containers using an arbitrarily assigned user ID.
This provides additional security against processes escaping the container due to a container
engine vulnerability and thereby achieving escalated permissions on the host node.
For an image to support running as an arbitrary user, directories and files that may be written
to by processes in the image should be owned by the root group and be read/writable by that group.
Files to be executed should also have group execute permissions.
Adding the following to your Dockerfile sets the directory and file permissions to allow users
in the root group to access them in the built image:
RUN chgrp -R 0 /some/directory && chmod -R g=u /some/directory
...
OpenShift does provide a way to run containers as root -- but we're absolutely trying to avoid that.
So it was that suggestion to add the 'chgrp' and 'chmod' to the Dockerfile that seemed to be the best and easiest solution.
> I believe our standard image is for building Drill.
> What you want is an image that uses an existing Drill build.
> The "SNAPSHOT" refers to the current master version of the code, which is probably not what you want.
> You want the released 1.17 binaries since you want to use, not develop, Drill.
Thank you for the suggestion. You were right, I was sitting on the master branch.
So I checked out the 1.17.0 branch.
Then I saw that there was no Dockerfile in this branch, so I copied the one I had been using (including the added chgrp/chmod), tried the 'docker build' -- and again got a similar build error regarding the parent pom.
But looking at the error, it seems to be more an issue with finding the specified Apache POM in the conjars.org/repo. The following simply does not exist:
http://conjars.org/repo/org/apache/apache/21/apache-21.pom
(And on a related sidenote, I've built the apache/camel project, and its parent POM has an essentially identical '<parent><groupId>org.apache' section as yours - and it doesn't barf with a "parent POM" problem - so that can't be the problem. It's obviously something to do with this connection to conjars.org.)
This is how I tried building the Drill Docker and the FATAL error I get:
% git branch
* (detached from origin/1.17.0)
master
% docker build -t mydrill .
Sending build context to Docker daemon 199.2MB
Step 1/10 : FROM maven:3.6-jdk-8 as build
---> 5042e76d4104
Step 2/10 : COPY . /src
---> 9a373ca8c131
Step 3/10 : WORKDIR /src
---> Running in 982bd7f24911
Removing intermediate container 982bd7f24911
---> 0b023e6084b1
Step 4/10 : RUN mvn clean install -DskipTests -q
---> Running in f2840c1bb274
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.17.0:
Could not transfer artifact org.apache:apache:pom:21 from/to conjars (http://conjars.org/repo):
Transfer failed for http://conjars.org/repo/org/apache/apache/21/apache-21.pom and
'parent.relativePath' points at no local POM @ line 24, column 11:
Connect to conjars.org:80 [conjars.org/184.73.255.175, conjars.org/54.225.137.155] failed:
Connection timed out (Connection timed out) -> [Help 2]
> Short term, the best solution would be if you can build an image based on the existing Drill image rather than making a copy of the Dockerfile.
Since I'm essentially new to Docker as well, how would I build a new image -- using the 1.17.0 image I already have in my local Docker repo (instead of building from source) -- and also doing the chgrp/chmod I need to do?
(I did figure out how to use 'docker save apache/drill > apache-drill.tar' to pull apart a container, and inspected all the layers to find the apache-drill-1.17.0.tar.gz you're referring to. But hopefully I don't need to go down *that* path of trying to build an image from scratch using your jars, and the 3rd party jars you include, etc, and hopefully there's a simpler way of building an image based on your image.)
Or maybe the solution to the conjars.org maven build error is trivial and I can try that again.
Thank you again so much!
Ron
> If we understand the original file permission problem, perhaps we can find a way to fix that.
> On January 29, 2020 at 2:10 AM Paul Rogers <pa...@yahoo.com.INVALID> wrote:
>
>
> Hi Ron,
>
> I don't think anyone on the Drill team has access to an OpenShift environment. Let's see if we can use your work to ensure that the Docker image supports OpenShift in the future.
>
> Please explain a bit more about the file permissions issue. Is the file owned by a user other than the one that runs Drill? If so, sounds like a bug, unless OpenShift uses a different user than plain Docker would.
>
>
> I believe our standard image is for building Drill. What you want is an image that uses an existing Drill build. The "SNAPSHOT" refers to the current master version of the code, which is probably not what you want. You want the released 1.17 binaries since you want to use, not develop, Drill.
>
> Question for the team: do we have a separate image for folks who want to run the latest 1.17 release?
>
> Short term, the best solution would be if you can build an image based on the existing Drill image rather than making a copy of the Dockerfile. If we understand the original file permission problem, perhaps we can find a way to fix that.
>
> Are you looking to run Drill in embedded mode (Sqlline in a container, you ssh into the container; config lost on Drill shutdown) or in server mode (config stored in ZK so it persists across container runs)?
>
>
> Thanks,
> - Paul
>
>
>
> On Tuesday, January 28, 2020, 9:41:28 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
>
>
> Hi, all. Drill and OpenShift newbie here.
>
> Has anyone successfully deployed a Drill Docker container to an OpenShift environment?
>
> While there is information about Drill Docker, there seems to be zero information about OpenShift in particular.
>
> Per the instructions at drill.apache.org/docs/running-drill-on-docker, I pulled the Drill Docker image from Docker Hub, and then pushed it to our OpenShift environment. But when I tried to deploy it, I immediately ran into an error about /opt/drill/conf/drill-override.conf not being readable.
>
> I understand why the problem is happening (because of who OpenShift runs the container as), so I downloaded the source from GitHub and modified the Dockerfile to include:
>
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> so that all of /opt/drill would be available to everyone. But then 'docker build' kept failing, giving the error:
>
> Non-resolvable parent POM for org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> Could not transfer artifact org.apache:apache:pom:21
>
> I tried researching that error but couldn't figure out what was going on. So I finally decided to start trying to mount persistent volumes, creating one PV for /opt/drill/conf (and then copying the default drill-override.conf there) and one PV for /opt/drill/log.
>
> Now the container gets much further, but eventually fails on something Hadoop related. I'm not trying to do anything with Hadoop, so I don't know what that's about, but it says I don't have HADOOP_HOME set.
>
> Hopefully I can figure out the remaining steps I need (an environment variable? more configs?), but I was wondering if anyone else had already successfully figured out how to deploy to OpenShift, or might know why the 'docker build' fails with that error?
>
> For what it's worth, I copied over only that drill-override.conf and nothing else. And I did not set any Drill environment variables in OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
>
> Thanks for any help!
>
> Ron
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Volodymyr Vysotskyi <vo...@apache.org>.
Hi all,
We have an image that includes the 1.17 release binaries:
apache/drill:1.17.0. It is a good point to try building a new image on top
of the existing one with the commands you need.
Kind regards,
Volodymyr Vysotskyi
On Wed, Jan 29, 2020 at 9:10 AM Paul Rogers <pa...@yahoo.com.invalid>
wrote:
> Hi Ron,
>
> I don't think anyone on the Drill team has access to an OpenShift
> environment. Let's see if we can use your work to ensure that the Docker
> image supports OpenShift in the future.
>
> Please explain a bit more about the file permissions issue. Is the file
> owned by a user other than the one that runs Drill? If so, sounds like a
> bug, unless OpenShift uses a different user than plain Docker would.
>
>
> I believe our standard image is for building Drill. What you want is an
> image that uses an existing Drill build. The "SNAPSHOT" refers to the
> current master version of the code, which is probably not what you want.
> You want the released 1.17 binaries since you want to use, not develop,
> Drill.
>
> Question for the team: do we have a separate image for folks who want to
> run the latest 1.17 release?
>
> Short term, the best solution would be if you can build an image based on
> the existing Drill image rather than making a copy of the Dockerfile. If we
> understand the original file permission problem, perhaps we can find a way
> to fix that.
>
> Are you looking to run Drill in embedded mode (Sqlline in a container, you
> ssh into the container; config lost on Drill shutdown) or in server mode
> (config stored in ZK so it persists across container runs)?
>
>
> Thanks,
> - Paul
>
>
>
> On Tuesday, January 28, 2020, 9:41:28 PM PST, Ron Cecchini <
> roncecchini@comcast.net> wrote:
>
>
> Hi, all. Drill and OpenShift newbie here.
>
> Has anyone successfully deployed a Drill Docker container to an OpenShift
> environment?
>
> While there is information about Drill Docker, there seems to be zero
> information about OpenShift in particular.
>
> Per the instructions at drill.apache.org/docs/running-drill-on-docker, I
> pulled the Drill Docker image from Docker Hub, and then pushed it to our
> OpenShift environment. But when I tried to deploy it, I immediately ran
> into an error about /opt/drill/conf/drill-override.conf not being readable.
>
> I understand why the problem is happening (because of who OpenShift runs
> the container as), so I downloaded the source from GitHub and modified the
> Dockerfile to include:
>
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> so that all of /opt/drill would be available to everyone. But then
> 'docker build' kept failing, giving the error:
>
> Non-resolvable parent POM for
> org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> Could not transfer artifact org.apache:apache:pom:21
>
> I tried researching that error but couldn't figure out what was going on.
> So I finally decided to start trying to mount persistent volumes, creating
> one PV for /opt/drill/conf (and then copying the default
> drill-override.conf there) and one PV for /opt/drill/log.
>
> Now the container gets much further, but eventually fails on something
> Hadoop related. I'm not trying to do anything with Hadoop, so I don't know
> what that's about, but it says I don't have HADOOP_HOME set.
>
> Hopefully I can figure out the remaining steps I need (an environment
> variable? more configs?), but I was wondering if anyone else had already
> successfully figured out how to deploy to OpenShift, or might know why the
> 'docker build' fails with that error?
>
> For what it's worth, I copied over only that drill-override.conf and
> nothing else. And I did not set any Drill environment variables in
> OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
>
> Thanks for any help!
>
> Ron
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Ron,
I don't think anyone on the Drill team has access to an OpenShift environment. Let's see if we can use your work to ensure that the Docker image supports OpenShift in the future.
Please explain a bit more about the file permissions issue. Is the file owned by a user other than the one that runs Drill? If so, sounds like a bug, unless OpenShift uses a different user than plain Docker would.
I believe our standard image is for building Drill. What you want is an image that uses an existing Drill build. The "SNAPSHOT" refers to the current master version of the code, which is probably not what you want. You want the released 1.17 binaries since you want to use, not develop, Drill.
Question for the team: do we have a separate image for folks who want to run the latest 1.17 release?
Short term, the best solution would be if you can build an image based on the existing Drill image rather than making a copy of the Dockerfile. If we understand the original file permission problem, perhaps we can find a way to fix that.
Are you looking to run Drill in embedded mode (Sqlline in a container, you ssh into the container; config lost on Drill shutdown) or in server mode (config stored in ZK so it persists across container runs)?
Thanks,
- Paul
On Tuesday, January 28, 2020, 9:41:28 PM PST, Ron Cecchini <ro...@comcast.net> wrote:
Hi, all. Drill and OpenShift newbie here.
Has anyone successfully deployed a Drill Docker container to an OpenShift environment?
While there is information about Drill Docker, there seems to be zero information about OpenShift in particular.
Per the instructions at drill.apache.org/docs/running-drill-on-docker, I pulled the Drill Docker image from Docker Hub, and then pushed it to our OpenShift environment. But when I tried to deploy it, I immediately ran into an error about /opt/drill/conf/drill-override.conf not being readable.
I understand why the problem is happening (because of who OpenShift runs the container as), so I downloaded the source from GitHub and modified the Dockerfile to include:
RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
so that all of /opt/drill would be available to everyone. But then 'docker build' kept failing, giving the error:
Non-resolvable parent POM for org.apache.drill:drill-root:1.18.0-SNAPSHOT:
Could not transfer artifact org.apache:apache:pom:21
I tried researching that error but couldn't figure out what was going on. So I finally decided to start trying to mount persistent volumes, creating one PV for /opt/drill/conf (and then copying the default drill-override.conf there) and one PV for /opt/drill/log.
Now the container gets much further, but eventually fails on something Hadoop related. I'm not trying to do anything with Hadoop, so I don't know what that's about, but it says I don't have HADOOP_HOME set.
Hopefully I can figure out the remaining steps I need (an environment variable? more configs?), but I was wondering if anyone else had already successfully figured out how to deploy to OpenShift, or might know why the 'docker build' fails with that error?
For what it's worth, I copied over only that drill-override.conf and nothing else. And I did not set any Drill environment variables in OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
Thanks for any help!
Ron
Re: Problem running Drill in a Docker container in OpenShift
Posted by Abhishek Girish <ag...@apache.org>.
Arina,
There is a v1.0 of the Drill Operator for Kubernetes out, but it is
currently tied to the MapR platform. Of the people here, Anton & Denys have
tried it out.
The open source version of it that I'm working on is based on that, minus
MapR integration and plus Apache integration. My plan is to get a preview
out as soon as possible. I don't have an ETA, but I think end of Feb is a
timeframe odd like to get something out.
Regards,
Abhishek
On Tue, Feb 4, 2020 at 6:06 PM Arina Yelchiyeva <ar...@gmail.com>
wrote:
> Abhishek, looks like Paul is planning to put on hold his efforts since he
> expects you open-source version to be out soon.
> Do you have an ETA?
> This work is important for the community and it would be nice if is
> completed in this release.
>
> Here Jira to tract the efforts:
> https://issues.apache.org/jira/browse/DRILL-7563 <
> https://issues.apache.org/jira/browse/DRILL-7563>
>
> Kind regards,
> Arina
>
> > On Feb 3, 2020, at 10:30 PM, Paul Rogers <pa...@yahoo.com.INVALID>
> wrote:
> >
> > Hi Abhishek,
> >
> > Thanks for the update! Seems to make sense to wait for you to open
> source your work than to spend time on duplicating your effort. And, people
> who want a solution short term can perhaps work with MapR/HPE as you
> suggest. Sounds like you have access to the various systems and have
> worked though the myriad details involved in creating a good integration.
> >
> > Does your work include Helm integration?
> >
> > The key challenge for any K8s integration is that Drill needs access to
> data, which requires some kind of distributed storage. This has long been a
> K8s weakness. But, it is, of course, a MapR strength.
> >
> > Please let us know if you need help with the open source efforts.
> >
> > Thanks,
> > - Paul
> >
> >
> >
> > On Monday, February 3, 2020, 3:13:28 AM PST, Abhishek Girish <
> agirish@apache.org> wrote:
> >
> > Hey Ron,
> >
> > As a part of MapR (now HPE), I've created a native operator for Apache
> > Drill and this works on multiple variants of Kubernetes including
> > OpenShift. With this, we introduce a new Kind called "DrillCluster" via a
> > Custom Resource Definition (CRD) and a Custom Controller (logic to manage
> > this DrillCluster kind - written in Golang) for the same. Using this,
> users
> > can easily deploy Drill clusters by submitting Custom Resource YAML files
> > (CRs) for the DrillCluster kind. It supports creation of multiple Drill
> > clusters (multiple Drillbits launched in distributed mode), multiple
> > versions (such as 1.15.0 and 1.16.0), auto-scaling the number of
> Drillbits
> > (based on CPU utilization) and more. I can share more details of this if
> > anyone's interested.
> >
> > While Vanilla K8S, and GKE worked out of the box, I had to make some
> > changes to support OpenShift (related to Service Accounts, Security
> Context
> > Constraints, etc). Perhaps you ran into similar issues (I'm yet to read
> > this thread fully).
> >
> > We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is
> > that the current release has dependencies and integrations with MapR's
> > distribution of Apache Drill and is close sourced at the moment (there is
> > plan to open source that in the near future).
> >
> > I have an open source variant of this in the works - to support vanilla
> > Apache Drill. In the current state, it has all similar features , it
> > removes the MapR specific integration (reliance on MapR-FS instead of
> HDFS,
> > MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper
> > integration instead. Let me know if you're interested - and I can share
> the
> > GitHub branch.
> >
> > Regards,
> > Abhishek
> >
> > [1]
> >
> https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/
> > [2]
> >
> https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html
> > [3] https://github.com/mapr/mapr-operators
> >
> > On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <ro...@comcast.net>
> > wrote:
> >
> >>
> >> Hi, all. Drill and OpenShift newbie here.
> >>
> >> Has anyone successfully deployed a Drill Docker container to an
> OpenShift
> >> environment?
> >>
> >> While there is information about Drill Docker, there seems to be zero
> >> information about OpenShift in particular.
> >>
> >> Per the instructions at drill.apache.org/docs/running-drill-on-docker,
> I
> >> pulled the Drill Docker image from Docker Hub, and then pushed it to our
> >> OpenShift environment. But when I tried to deploy it, I immediately ran
> >> into an error about /opt/drill/conf/drill-override.conf not being
> readable.
> >>
> >> I understand why the problem is happening (because of who OpenShift runs
> >> the container as), so I downloaded the source from GitHub and modified
> the
> >> Dockerfile to include:
> >>
> >> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
> >>
> >> so that all of /opt/drill would be available to everyone. But then
> >> 'docker build' kept failing, giving the error:
> >>
> >> Non-resolvable parent POM for
> >> org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> >> Could not transfer artifact org.apache:apache:pom:21
> >>
> >> I tried researching that error but couldn't figure out what was going
> on.
> >> So I finally decided to start trying to mount persistent volumes,
> creating
> >> one PV for /opt/drill/conf (and then copying the default
> >> drill-override.conf there) and one PV for /opt/drill/log.
> >>
> >> Now the container gets much further, but eventually fails on something
> >> Hadoop related. I'm not trying to do anything with Hadoop, so I don't
> know
> >> what that's about, but it says I don't have HADOOP_HOME set.
> >>
> >> Hopefully I can figure out the remaining steps I need (an environment
> >> variable? more configs?), but I was wondering if anyone else had
> already
> >> successfully figured out how to deploy to OpenShift, or might know why
> the
> >> 'docker build' fails with that error?
> >>
> >> For what it's worth, I copied over only that drill-override.conf and
> >> nothing else. And I did not set any Drill environment variables in
> >> OpenShift. I'm basically trying to run the "vanilla" Drill Docker
> as-is.
> >>
> >> Thanks for any help!
> >>
> >> Ron
> >>
>
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Arina Yelchiyeva <ar...@gmail.com>.
Abhishek, looks like Paul is planning to put on hold his efforts since he expects you open-source version to be out soon.
Do you have an ETA?
This work is important for the community and it would be nice if is completed in this release.
Here Jira to tract the efforts: https://issues.apache.org/jira/browse/DRILL-7563 <https://issues.apache.org/jira/browse/DRILL-7563>
Kind regards,
Arina
> On Feb 3, 2020, at 10:30 PM, Paul Rogers <pa...@yahoo.com.INVALID> wrote:
>
> Hi Abhishek,
>
> Thanks for the update! Seems to make sense to wait for you to open source your work than to spend time on duplicating your effort. And, people who want a solution short term can perhaps work with MapR/HPE as you suggest. Sounds like you have access to the various systems and have worked though the myriad details involved in creating a good integration.
>
> Does your work include Helm integration?
>
> The key challenge for any K8s integration is that Drill needs access to data, which requires some kind of distributed storage. This has long been a K8s weakness. But, it is, of course, a MapR strength.
>
> Please let us know if you need help with the open source efforts.
>
> Thanks,
> - Paul
>
>
>
> On Monday, February 3, 2020, 3:13:28 AM PST, Abhishek Girish <ag...@apache.org> wrote:
>
> Hey Ron,
>
> As a part of MapR (now HPE), I've created a native operator for Apache
> Drill and this works on multiple variants of Kubernetes including
> OpenShift. With this, we introduce a new Kind called "DrillCluster" via a
> Custom Resource Definition (CRD) and a Custom Controller (logic to manage
> this DrillCluster kind - written in Golang) for the same. Using this, users
> can easily deploy Drill clusters by submitting Custom Resource YAML files
> (CRs) for the DrillCluster kind. It supports creation of multiple Drill
> clusters (multiple Drillbits launched in distributed mode), multiple
> versions (such as 1.15.0 and 1.16.0), auto-scaling the number of Drillbits
> (based on CPU utilization) and more. I can share more details of this if
> anyone's interested.
>
> While Vanilla K8S, and GKE worked out of the box, I had to make some
> changes to support OpenShift (related to Service Accounts, Security Context
> Constraints, etc). Perhaps you ran into similar issues (I'm yet to read
> this thread fully).
>
> We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is
> that the current release has dependencies and integrations with MapR's
> distribution of Apache Drill and is close sourced at the moment (there is
> plan to open source that in the near future).
>
> I have an open source variant of this in the works - to support vanilla
> Apache Drill. In the current state, it has all similar features , it
> removes the MapR specific integration (reliance on MapR-FS instead of HDFS,
> MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper
> integration instead. Let me know if you're interested - and I can share the
> GitHub branch.
>
> Regards,
> Abhishek
>
> [1]
> https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/
> [2]
> https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html
> [3] https://github.com/mapr/mapr-operators
>
> On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <ro...@comcast.net>
> wrote:
>
>>
>> Hi, all. Drill and OpenShift newbie here.
>>
>> Has anyone successfully deployed a Drill Docker container to an OpenShift
>> environment?
>>
>> While there is information about Drill Docker, there seems to be zero
>> information about OpenShift in particular.
>>
>> Per the instructions at drill.apache.org/docs/running-drill-on-docker, I
>> pulled the Drill Docker image from Docker Hub, and then pushed it to our
>> OpenShift environment. But when I tried to deploy it, I immediately ran
>> into an error about /opt/drill/conf/drill-override.conf not being readable.
>>
>> I understand why the problem is happening (because of who OpenShift runs
>> the container as), so I downloaded the source from GitHub and modified the
>> Dockerfile to include:
>>
>> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>>
>> so that all of /opt/drill would be available to everyone. But then
>> 'docker build' kept failing, giving the error:
>>
>> Non-resolvable parent POM for
>> org.apache.drill:drill-root:1.18.0-SNAPSHOT:
>> Could not transfer artifact org.apache:apache:pom:21
>>
>> I tried researching that error but couldn't figure out what was going on.
>> So I finally decided to start trying to mount persistent volumes, creating
>> one PV for /opt/drill/conf (and then copying the default
>> drill-override.conf there) and one PV for /opt/drill/log.
>>
>> Now the container gets much further, but eventually fails on something
>> Hadoop related. I'm not trying to do anything with Hadoop, so I don't know
>> what that's about, but it says I don't have HADOOP_HOME set.
>>
>> Hopefully I can figure out the remaining steps I need (an environment
>> variable? more configs?), but I was wondering if anyone else had already
>> successfully figured out how to deploy to OpenShift, or might know why the
>> 'docker build' fails with that error?
>>
>> For what it's worth, I copied over only that drill-override.conf and
>> nothing else. And I did not set any Drill environment variables in
>> OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
>>
>> Thanks for any help!
>>
>> Ron
>>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Abhishek Girish <ag...@apache.org>.
Hey Paul,
Sorry for my delayed response. And thanks for your encouragement.
So here's a brief history of my work in this area.
(1) I started with simple YAML based deployments for Drill. Used standard
Kubernetes APIs and Controllers. This supported bringing up Drill in
distributed mode, used MapR ZK for Cluster coordination and had a MapR
client to connect to MapR FS. It had standard features such as resizing and
such. So pretty much basic Drill on Kubernetes use cases were supported.
(2) I then added Helm charts support to make (1) more easy to use.
(3) I began parallel effort to build an Operator for Drill (written in go).
This would create a Custom Resource called DrillCluster - so in YAML files
you would see Kind as "DrillCluster" instead of Pod / Statefulset / or
similar. The operator model in K8S is now seeing more adoption as it is
more powerful, flexible and simpler for users. For instance, I could add
more code checks and validations, logging for debugging & more when
compared to approaches (1) and (2). Also has more potential for adding more
features and fixes in this model. This is what we shipped at MapR and this
is what I'm working on for open source Drill and planning on sharing soon.
What's pending for an initial preview release is replacing MapR client with
say HDFS client and MapR ZK with Apache ZK. I can also share (1) and (2)
soon after that.
I'll definitely count on your vast experience and knowledge for help in
this regard.
Regards,
Abhishek
On Tue, Feb 4, 2020 at 2:00 AM Paul Rogers <pa...@yahoo.com.invalid>
wrote:
> Hi Abhishek,
>
> Thanks for the update! Seems to make sense to wait for you to open source
> your work than to spend time on duplicating your effort. And, people who
> want a solution short term can perhaps work with MapR/HPE as you suggest.
> Sounds like you have access to the various systems and have worked though
> the myriad details involved in creating a good integration.
>
> Does your work include Helm integration?
>
> The key challenge for any K8s integration is that Drill needs access to
> data, which requires some kind of distributed storage. This has long been a
> K8s weakness. But, it is, of course, a MapR strength.
>
> Please let us know if you need help with the open source efforts.
>
> Thanks,
> - Paul
>
>
>
> On Monday, February 3, 2020, 3:13:28 AM PST, Abhishek Girish <
> agirish@apache.org> wrote:
>
> Hey Ron,
>
> As a part of MapR (now HPE), I've created a native operator for Apache
> Drill and this works on multiple variants of Kubernetes including
> OpenShift. With this, we introduce a new Kind called "DrillCluster" via a
> Custom Resource Definition (CRD) and a Custom Controller (logic to manage
> this DrillCluster kind - written in Golang) for the same. Using this, users
> can easily deploy Drill clusters by submitting Custom Resource YAML files
> (CRs) for the DrillCluster kind. It supports creation of multiple Drill
> clusters (multiple Drillbits launched in distributed mode), multiple
> versions (such as 1.15.0 and 1.16.0), auto-scaling the number of Drillbits
> (based on CPU utilization) and more. I can share more details of this if
> anyone's interested.
>
> While Vanilla K8S, and GKE worked out of the box, I had to make some
> changes to support OpenShift (related to Service Accounts, Security Context
> Constraints, etc). Perhaps you ran into similar issues (I'm yet to read
> this thread fully).
>
> We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is
> that the current release has dependencies and integrations with MapR's
> distribution of Apache Drill and is close sourced at the moment (there is
> plan to open source that in the near future).
>
> I have an open source variant of this in the works - to support vanilla
> Apache Drill. In the current state, it has all similar features , it
> removes the MapR specific integration (reliance on MapR-FS instead of HDFS,
> MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper
> integration instead. Let me know if you're interested - and I can share the
> GitHub branch.
>
> Regards,
> Abhishek
>
> [1]
>
> https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/
> [2]
>
> https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html
> [3] https://github.com/mapr/mapr-operators
>
> On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <ro...@comcast.net>
> wrote:
>
> >
> > Hi, all. Drill and OpenShift newbie here.
> >
> > Has anyone successfully deployed a Drill Docker container to an OpenShift
> > environment?
> >
> > While there is information about Drill Docker, there seems to be zero
> > information about OpenShift in particular.
> >
> > Per the instructions at drill.apache.org/docs/running-drill-on-docker, I
> > pulled the Drill Docker image from Docker Hub, and then pushed it to our
> > OpenShift environment. But when I tried to deploy it, I immediately ran
> > into an error about /opt/drill/conf/drill-override.conf not being
> readable.
> >
> > I understand why the problem is happening (because of who OpenShift runs
> > the container as), so I downloaded the source from GitHub and modified
> the
> > Dockerfile to include:
> >
> > RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
> >
> > so that all of /opt/drill would be available to everyone. But then
> > 'docker build' kept failing, giving the error:
> >
> > Non-resolvable parent POM for
> > org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> > Could not transfer artifact org.apache:apache:pom:21
> >
> > I tried researching that error but couldn't figure out what was going on.
> > So I finally decided to start trying to mount persistent volumes,
> creating
> > one PV for /opt/drill/conf (and then copying the default
> > drill-override.conf there) and one PV for /opt/drill/log.
> >
> > Now the container gets much further, but eventually fails on something
> > Hadoop related. I'm not trying to do anything with Hadoop, so I don't
> know
> > what that's about, but it says I don't have HADOOP_HOME set.
> >
> > Hopefully I can figure out the remaining steps I need (an environment
> > variable? more configs?), but I was wondering if anyone else had already
> > successfully figured out how to deploy to OpenShift, or might know why
> the
> > 'docker build' fails with that error?
> >
> > For what it's worth, I copied over only that drill-override.conf and
> > nothing else. And I did not set any Drill environment variables in
> > OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
> >
> > Thanks for any help!
> >
> > Ron
> >
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Paul Rogers <pa...@yahoo.com.INVALID>.
Hi Abhishek,
Thanks for the update! Seems to make sense to wait for you to open source your work than to spend time on duplicating your effort. And, people who want a solution short term can perhaps work with MapR/HPE as you suggest. Sounds like you have access to the various systems and have worked though the myriad details involved in creating a good integration.
Does your work include Helm integration?
The key challenge for any K8s integration is that Drill needs access to data, which requires some kind of distributed storage. This has long been a K8s weakness. But, it is, of course, a MapR strength.
Please let us know if you need help with the open source efforts.
Thanks,
- Paul
On Monday, February 3, 2020, 3:13:28 AM PST, Abhishek Girish <ag...@apache.org> wrote:
Hey Ron,
As a part of MapR (now HPE), I've created a native operator for Apache
Drill and this works on multiple variants of Kubernetes including
OpenShift. With this, we introduce a new Kind called "DrillCluster" via a
Custom Resource Definition (CRD) and a Custom Controller (logic to manage
this DrillCluster kind - written in Golang) for the same. Using this, users
can easily deploy Drill clusters by submitting Custom Resource YAML files
(CRs) for the DrillCluster kind. It supports creation of multiple Drill
clusters (multiple Drillbits launched in distributed mode), multiple
versions (such as 1.15.0 and 1.16.0), auto-scaling the number of Drillbits
(based on CPU utilization) and more. I can share more details of this if
anyone's interested.
While Vanilla K8S, and GKE worked out of the box, I had to make some
changes to support OpenShift (related to Service Accounts, Security Context
Constraints, etc). Perhaps you ran into similar issues (I'm yet to read
this thread fully).
We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is
that the current release has dependencies and integrations with MapR's
distribution of Apache Drill and is close sourced at the moment (there is
plan to open source that in the near future).
I have an open source variant of this in the works - to support vanilla
Apache Drill. In the current state, it has all similar features , it
removes the MapR specific integration (reliance on MapR-FS instead of HDFS,
MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper
integration instead. Let me know if you're interested - and I can share the
GitHub branch.
Regards,
Abhishek
[1]
https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/
[2]
https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html
[3] https://github.com/mapr/mapr-operators
On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <ro...@comcast.net>
wrote:
>
> Hi, all. Drill and OpenShift newbie here.
>
> Has anyone successfully deployed a Drill Docker container to an OpenShift
> environment?
>
> While there is information about Drill Docker, there seems to be zero
> information about OpenShift in particular.
>
> Per the instructions at drill.apache.org/docs/running-drill-on-docker, I
> pulled the Drill Docker image from Docker Hub, and then pushed it to our
> OpenShift environment. But when I tried to deploy it, I immediately ran
> into an error about /opt/drill/conf/drill-override.conf not being readable.
>
> I understand why the problem is happening (because of who OpenShift runs
> the container as), so I downloaded the source from GitHub and modified the
> Dockerfile to include:
>
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> so that all of /opt/drill would be available to everyone. But then
> 'docker build' kept failing, giving the error:
>
> Non-resolvable parent POM for
> org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> Could not transfer artifact org.apache:apache:pom:21
>
> I tried researching that error but couldn't figure out what was going on.
> So I finally decided to start trying to mount persistent volumes, creating
> one PV for /opt/drill/conf (and then copying the default
> drill-override.conf there) and one PV for /opt/drill/log.
>
> Now the container gets much further, but eventually fails on something
> Hadoop related. I'm not trying to do anything with Hadoop, so I don't know
> what that's about, but it says I don't have HADOOP_HOME set.
>
> Hopefully I can figure out the remaining steps I need (an environment
> variable? more configs?), but I was wondering if anyone else had already
> successfully figured out how to deploy to OpenShift, or might know why the
> 'docker build' fails with that error?
>
> For what it's worth, I copied over only that drill-override.conf and
> nothing else. And I did not set any Drill environment variables in
> OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
>
> Thanks for any help!
>
> Ron
>
Re: Problem running Drill in a Docker container in OpenShift
Posted by Abhishek Girish <ag...@apache.org>.
Hey Ron,
As a part of MapR (now HPE), I've created a native operator for Apache
Drill and this works on multiple variants of Kubernetes including
OpenShift. With this, we introduce a new Kind called "DrillCluster" via a
Custom Resource Definition (CRD) and a Custom Controller (logic to manage
this DrillCluster kind - written in Golang) for the same. Using this, users
can easily deploy Drill clusters by submitting Custom Resource YAML files
(CRs) for the DrillCluster kind. It supports creation of multiple Drill
clusters (multiple Drillbits launched in distributed mode), multiple
versions (such as 1.15.0 and 1.16.0), auto-scaling the number of Drillbits
(based on CPU utilization) and more. I can share more details of this if
anyone's interested.
While Vanilla K8S, and GKE worked out of the box, I had to make some
changes to support OpenShift (related to Service Accounts, Security Context
Constraints, etc). Perhaps you ran into similar issues (I'm yet to read
this thread fully).
We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is
that the current release has dependencies and integrations with MapR's
distribution of Apache Drill and is close sourced at the moment (there is
plan to open source that in the near future).
I have an open source variant of this in the works - to support vanilla
Apache Drill. In the current state, it has all similar features , it
removes the MapR specific integration (reliance on MapR-FS instead of HDFS,
MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper
integration instead. Let me know if you're interested - and I can share the
GitHub branch.
Regards,
Abhishek
[1]
https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/
[2]
https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html
[3] https://github.com/mapr/mapr-operators
On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <ro...@comcast.net>
wrote:
>
> Hi, all. Drill and OpenShift newbie here.
>
> Has anyone successfully deployed a Drill Docker container to an OpenShift
> environment?
>
> While there is information about Drill Docker, there seems to be zero
> information about OpenShift in particular.
>
> Per the instructions at drill.apache.org/docs/running-drill-on-docker, I
> pulled the Drill Docker image from Docker Hub, and then pushed it to our
> OpenShift environment. But when I tried to deploy it, I immediately ran
> into an error about /opt/drill/conf/drill-override.conf not being readable.
>
> I understand why the problem is happening (because of who OpenShift runs
> the container as), so I downloaded the source from GitHub and modified the
> Dockerfile to include:
>
> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill
>
> so that all of /opt/drill would be available to everyone. But then
> 'docker build' kept failing, giving the error:
>
> Non-resolvable parent POM for
> org.apache.drill:drill-root:1.18.0-SNAPSHOT:
> Could not transfer artifact org.apache:apache:pom:21
>
> I tried researching that error but couldn't figure out what was going on.
> So I finally decided to start trying to mount persistent volumes, creating
> one PV for /opt/drill/conf (and then copying the default
> drill-override.conf there) and one PV for /opt/drill/log.
>
> Now the container gets much further, but eventually fails on something
> Hadoop related. I'm not trying to do anything with Hadoop, so I don't know
> what that's about, but it says I don't have HADOOP_HOME set.
>
> Hopefully I can figure out the remaining steps I need (an environment
> variable? more configs?), but I was wondering if anyone else had already
> successfully figured out how to deploy to OpenShift, or might know why the
> 'docker build' fails with that error?
>
> For what it's worth, I copied over only that drill-override.conf and
> nothing else. And I did not set any Drill environment variables in
> OpenShift. I'm basically trying to run the "vanilla" Drill Docker as-is.
>
> Thanks for any help!
>
> Ron
>