You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by cfries <gi...@git.apache.org> on 2017/04/27 17:41:03 UTC
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
GitHub user cfries opened a pull request:
https://github.com/apache/zeppelin/pull/2294
[ZEPPELIN-2438]: Add local winutils.exe in build step and use if required \u2026
### What is this PR for?
Add local winutils.exe in build and use as temporary HADOOP_HOME if required on run (windows only).
The build of the spark dependencies now downloads winutils.exe and installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if HADOOP_HOME exists, if not, it will set it to point to winutils. This is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed from the .gitignore.
### What type of PR is it?
[Bug Fix | Improvement]
Without this fix, the binary distribution or a local build of Zeppelin will have the spark interpreter (default interpreter) fail on a Windows platform (unless the user follows additional / manual installation steps or already has a local HADOOP). This will frustrate users trying to checkout Zeppelin.
For users having HADOOP or macOS/Linux users, the chance will have no effect.
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2438
### How should this be tested?
- Non-Regression Test
- Standard Unit Test
- Run Zeppelin without additional installations (except Java) on Windows
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cfries/zeppelin windows-compatiblity
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2294
----
commit 303511f15dc95b0b75b224607f79ef00229392ba
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-27T17:19:28Z
[ZEPPELIN-2438]: Add local winutils.exe in build and use if required on run.
The build of the spark dependencies now downloads winutils.exe and
installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if
HADOOP_HOME exists, if not, it will set it to point to winutils. This
is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed
from the .gitignore.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
Posted by cfries <gi...@git.apache.org>.
Github user cfries closed the pull request at:
https://github.com/apache/zeppelin/pull/2294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2294: [ZEPPELIN-2438]: Add local winutils.exe in build step ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2294
Again, I'm not sure about releasing with a binary embedded in resource and how that correspond to the Apache release process.
But, I understand where you are coming up, and if you have a way to bootstrap the download only on the Windows platform, we could take a look at that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2294: [ZEPPELIN-2438]: Add local winutils.exe in build step ...
Posted by cfries <gi...@git.apache.org>.
Github user cfries commented on the issue:
https://github.com/apache/zeppelin/pull/2294
@felixcheung I don't know if it would be an issue embedding and .exe binary in the binary distribution. If that *is* the case, the either
a) default-dependency on hadoop should be removed
or
b) documentation should be added and - if possible - a dialog prompting the used for the download (s.th. "This Spark interpreter is pre-configures for Hadoop and requires winutils.exe. ....")
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2294: [ZEPPELIN-2438]: Add local winutils.exe in build step ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2294
I think it's intentional that we ship a template of conf/zeppelin-env.cmd.template?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
Posted by cfries <gi...@git.apache.org>.
Github user cfries closed the pull request at:
https://github.com/apache/zeppelin/pull/2294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
Posted by cfries <gi...@git.apache.org>.
GitHub user cfries reopened a pull request:
https://github.com/apache/zeppelin/pull/2294
[ZEPPELIN-2438]: Add local winutils.exe in build step and use if required …
### What is this PR for?
Add local winutils.exe in build and use as temporary HADOOP_HOME if required on run (windows only).
The build of the spark dependencies now downloads winutils.exe and installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if HADOOP_HOME exists, if not, it will set it to point to winutils. This is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed from the .gitignore.
### What type of PR is it?
[Bug Fix | Improvement]
Without this fix, the binary distribution or a local build of Zeppelin will have the spark interpreter (default interpreter) fail on a Windows platform (unless the user follows additional / manual installation steps or already has a local HADOOP). This will frustrate users trying to checkout Zeppelin.
For users having HADOOP or macOS/Linux users, the chance will have no effect.
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2438
### How should this be tested?
- Non-Regression Test
- Standard Unit Test
- Run Zeppelin without additional installations (except Java) on Windows
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cfries/zeppelin windows-compatiblity
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2294
----
commit 3635ed8a717108ba82753413bd0500519ffd4add
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-08T20:00:33Z
[ZEPPELIN-2375]: Avoid modification of CLASSPATH variable.
The script previously modified the CLASSPATH variable. This may lead to
an undesired side effect, where the zeppelin server classpath is
exported to the zeppelin interpreter classpath, see JIRA issue
ZEPPELIN-2375 for details. Instead of modifying classpath we now work
the other way around and set ZEPPELIN_CLASSPATH or
ZEPPELIN_INT_CLASSPATH to include CLASSPATH and then use
ZEPPELIN_CLASSPATH or ZEPPELIN_INT_CLASSPATH in the corresponding exec
(runner), respectively.
See also https://issues.apache.org/jira/browse/ZEPPELIN-2375
commit 096fed6cfde37051b95018cedd455db3084aa1c9
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-27T16:38:53Z
Merge branch 'master' into apache/master
commit 303511f15dc95b0b75b224607f79ef00229392ba
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-27T17:19:28Z
[ZEPPELIN-2438]: Add local winutils.exe in build and use if required on run.
The build of the spark dependencies now downloads winutils.exe and
installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if
HADOOP_HOME exists, if not, it will set it to point to winutils. This
is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed
from the .gitignore.
commit 18e226be7dfc11493edd17bf039019bb0da93e15
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-30T00:11:21Z
Added zeppelin-env.cmd to distribution.
commit 4ba7f564841cb09072969f3538460a2ff285fbcf
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T10:43:17Z
Moved fallback setting HADOOP_HOME to common.cmd.
commit 97f97850d2fb1e2056d99530c78d754f02841a67
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T10:45:31Z
Cosmetic change. Added newline to end (effectively reverting previous commit).
commit 93017a036cba2b49e00763f8ac65d977c9ccab26
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T11:23:42Z
Moved fallback setting HADOOP_HOME to common.cmd.
Fixed syntax and path.
commit f59e39eedda069c29f0c1945f39e6e8feb259afc
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T14:13:57Z
Merge branch 'apache/master' into windows-compatiblity
commit b4babef9add9c1d6dfda39b3fdec032a76d9d26c
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-05T17:36:45Z
Added remark on additional Zepplin Spark on Windows requirements.
Zeppelin Spark on Windows requires winutils.exe which requires the
Visual Studio C++ Runtime. A remark clarifies this.
commit 84796708806410c35f37ba01e35eb45d74644133
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-05T17:36:45Z
Added remark on additional Zepplin Spark on Windows requirements.
Zeppelin Spark on Windows requires winutils.exe which requires the
Visual Studio C++ Runtime. A remark clarifies this.
(Added link).
commit 5e30b3c5c70a87701a6c6643f396b9a6fe35cb94
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-05T19:32:28Z
Merge remote-tracking branch 'origin/windows-compatiblity' into
windows-compatiblity
# Conflicts:
# docs/install/install.md
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2294: [ZEPPELIN-2438]: Add local winutils.exe in build step ...
Posted by cfries <gi...@git.apache.org>.
Github user cfries commented on the issue:
https://github.com/apache/zeppelin/pull/2294
(close and reopen to kick Jenkins to recognise Travis)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
Posted by cfries <gi...@git.apache.org>.
GitHub user cfries reopened a pull request:
https://github.com/apache/zeppelin/pull/2294
[ZEPPELIN-2438]: Add local winutils.exe in build step and use if required …
### What is this PR for?
Add local winutils.exe in build and use as temporary HADOOP_HOME if required on run (windows only).
The build of the spark dependencies now downloads winutils.exe and installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if HADOOP_HOME exists, if not, it will set it to point to winutils. This is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed from the .gitignore.
### What type of PR is it?
[Bug Fix | Improvement]
Without this fix, the binary distribution or a local build of Zeppelin will have the spark interpreter (default interpreter) fail on a Windows platform (unless the user follows additional / manual installation steps or already has a local HADOOP). This will frustrate users trying to checkout Zeppelin.
For users having HADOOP or macOS/Linux users, the chance will have no effect.
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2438
### How should this be tested?
- Non-Regression Test
- Standard Unit Test
- Run Zeppelin without additional installations (except Java) on Windows
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cfries/zeppelin windows-compatiblity
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2294
----
commit 3635ed8a717108ba82753413bd0500519ffd4add
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-08T20:00:33Z
[ZEPPELIN-2375]: Avoid modification of CLASSPATH variable.
The script previously modified the CLASSPATH variable. This may lead to
an undesired side effect, where the zeppelin server classpath is
exported to the zeppelin interpreter classpath, see JIRA issue
ZEPPELIN-2375 for details. Instead of modifying classpath we now work
the other way around and set ZEPPELIN_CLASSPATH or
ZEPPELIN_INT_CLASSPATH to include CLASSPATH and then use
ZEPPELIN_CLASSPATH or ZEPPELIN_INT_CLASSPATH in the corresponding exec
(runner), respectively.
See also https://issues.apache.org/jira/browse/ZEPPELIN-2375
commit 096fed6cfde37051b95018cedd455db3084aa1c9
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-27T16:38:53Z
Merge branch 'master' into apache/master
commit 303511f15dc95b0b75b224607f79ef00229392ba
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-27T17:19:28Z
[ZEPPELIN-2438]: Add local winutils.exe in build and use if required on run.
The build of the spark dependencies now downloads winutils.exe and
installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if
HADOOP_HOME exists, if not, it will set it to point to winutils. This
is required to have the spark interpreter work in Windows.
In order to have this active, the zeppelin-env.cmd had to be removed
from the .gitignore.
commit 18e226be7dfc11493edd17bf039019bb0da93e15
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-04-30T00:11:21Z
Added zeppelin-env.cmd to distribution.
commit 4ba7f564841cb09072969f3538460a2ff285fbcf
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T10:43:17Z
Moved fallback setting HADOOP_HOME to common.cmd.
commit 97f97850d2fb1e2056d99530c78d754f02841a67
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T10:45:31Z
Cosmetic change. Added newline to end (effectively reverting previous commit).
commit 93017a036cba2b49e00763f8ac65d977c9ccab26
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T11:23:42Z
Moved fallback setting HADOOP_HOME to common.cmd.
Fixed syntax and path.
commit f59e39eedda069c29f0c1945f39e6e8feb259afc
Author: Christian Fries <em...@christian-fries.de>
Date: 2017-05-01T14:13:57Z
Merge branch 'apache/master' into windows-compatiblity
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2294
---