You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by paul-rogers <gi...@git.apache.org> on 2017/10/26 16:47:30 UTC

[GitHub] drill pull request #1011: Drill 1170: Drill-on-YARN

GitHub user paul-rogers opened a pull request:

    https://github.com/apache/drill/pull/1011

    Drill 1170: Drill-on-YARN

    Provides Drill integration with YARN. Runs Drill as a long-running task under YARN. Monitors the Drill cluster, restarting failed Drillbits. Provides a command-line UI to start, stop and resize the cluster. Provides a web-based UI to monitor the cluster.
    
    The Drill-on-YARN (DoY) code has been in use by commercial users for over a year, since Drill 1.8 and has proven quite stable. Usage has been on MapR's version of YARN, we seek feedback from users of the Apache and other versions of YARN.
    
    See [DRILL-1170](https://issues.apache.org/jira/browse/DRILL-1170) for design information. See the included `README.md`` for internals information and `USAGE.md` for a detailed user guide.
    
    This is a large PR; it will take time to review. The key goal at this moment is to allow interested users to download the PR, build DoY, and try it out in their environments. The DoY code is mostly independent of Drill itself. The DoY code can be used to launch any version of Drill since 1.8. See the usage guide for information.
    
    It has been suggested that the code move to the `contrib` directory. That change will be made. But, since the code works successfully in its current location; we'll leave it their for now to ensure users are successful if they choose to try it.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/paul-rogers/drill DRILL-1170

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1011
    
----
commit e26012bc56cad3bf2819dff3bbdf70664de34955
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:24:00Z

    DRILL-1170: YARN integration for Drill
    
    This commit includes documentation files.

commit 3a6ffe78d9fe0e9a5beacd100e2e0ee6b40c7f34
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:25:34Z

    Client app

commit 509410c9710fc1ff23a4a13f51320a4c153f9328
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:26:50Z

    Files common to several modules

commit 36b8d323118077115ef1097ba8b23f6fc4a5390a
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:42:17Z

    Application master

commit 7fbc387634bb1acaf7807b02348b40364c81d282
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:44:33Z

    App Master web UI

commit 21fb93792290625b719899a4742573b8c3d4a7ce
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:45:44Z

    Distribution and project files

commit 567d36787b9ada60dd2141077e629158c53fc0c4
Author: Paul Rogers <pr...@maprtech.com>
Date:   2017-10-26T07:47:54Z

    Test files

----


---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

Posted by Agirish <gi...@git.apache.org>.

Github user Agirish commented on the issue:

    https://github.com/apache/drill/pull/1011
  
    @paul-rogers, I'll give it a try and update with my findings. 


---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

Posted by kr-arjun <gi...@git.apache.org>.

Github user kr-arjun commented on the issue:

    https://github.com/apache/drill/pull/1011
  
    @paul-rogers  I was able to resolve this issue by workaround of setting 'yarn.timeline-service.enabled' to false ( Copied yarn-site.xml with this property set to $DRILL_SITE directory). 
    
    This issue is specific to environment where Timeline server is enabled. Initially , it failed with 'java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig'. On copying required jars to Drill classpath , it failed with exception I have shared in the previous attachment. The same issue is reported in Spark as well (https://issues.apache.org/jira/browse/SPARK-15343). To find the error stack trace, I had to modify the DrillOnYarn.java to print StackTrace. Thought it would be useful if stack trace can be logged for troubleshooting purpose.


---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

Posted by paul-rogers <gi...@git.apache.org>.

Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1011
  
    Failing in Travis, apparently due to test-framework issue:
    ```
    Caused by: java.lang.ClassNotFoundException: org.apache.drill.categories.SecurityTest
    ```
    
    @ilooner, any idea what's going on? 


---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

Posted by paul-rogers <gi...@git.apache.org>.

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/1011

@kr-arjun, thanks for your note on error handling. Where you using the `start` command? There is exactly one place where the error "Failed to start Drill application master" is thrown: it is when Drill-on-YARN fails to start the application master. There are lots of other messages for other issues such as "Error: AM already running as Application ID: 1234" or "Failed to allocate Drill application master."

When writing the client, I made an explicit decision not to create a log file to avoid cluttering up things. There is no good place to put a client log since Drill does not actually run on the client machine. We could add a log, but it would be messy.

What we can do, however, is include the text of the message we got from YARN when we tried to start the AM process. When an error occurs, the client will now print something line the following:

```
Failed to start Drill application master.
Caused by: Some YARN error
```

The other thing we can add is a full stack dump, but only when requested with the `-v` (verbose) option:

```
> drill-on-yarn.sh -v start
Failed to start Drill application master.
Caused by: Some YARN error
Full stack trace:
(stack trace here)
```

I can't easily test this code. Please grab the latest sources and rerun your test case to ensure that it now prints out more information: whatever YARN tells us about why it would not start the AM.

---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

Posted by ilooner <gi...@git.apache.org>.

Github user ilooner commented on the issue:

    https://github.com/apache/drill/pull/1011
  
    @paul-rogers You need to add this dependency to your drill-yarn pom.xml
    
    ```
        <dependency>
          <groupId>org.apache.drill</groupId>
          <artifactId>drill-common</artifactId>
          <version>${project.version}</version>
          <classifier>tests</classifier>
          <scope>test</scope>
        </dependency>
    ```


---