You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/04/12 15:56:07 UTC
[GitHub] [solr] gerlowskija opened a new pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
gerlowskija opened a new pull request #74:
URL: https://github.com/apache/solr/pull/74
# Description
With the expansion of its streaming and math expression libraries, Solr now has a lot of analytics capabilities that integrate well with visualization and notebook software. But Solr doesn't offer anything out of the box to aid in integration with popular notebook and visualization tools. Providing an integration point with Apache Zeppelin, a popular notebook option, would go a long way to enabling analytics users to get started with Solr.
# Solution
This PR create a "ZeppelinTool" integrated into the "bin/solr" scripts. The tool deploys Zeppelin into a local sandbox and then bootstraps it to talk to Solr. Other cleanup and update commands are supported.
# Tests
"bin/solr" scripts don't lend themselves to testing under Solr's test suites, so this has had to get by with manual testing on Linux and Mac. (Windows to come shortly.)
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
gerlowskija commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-819722432
Hey @HoustonPutman Thanks for chiming in. A few replies inline.
> what does this provide over running Zeppelin locally yourself?
It automates all of the needed setup behind a single command: `bin/solr zeppelin bootstrap`. So in a word: convenience. You're right that Solr doesn't "know" about Zeppelin the same way that it knows about ZK, though Zeppelin isn't a totally random tool either. Our ["Math Expressions" docs](https://solr.apache.org/guide/8_8/math-expressions.html) make heavy use of Zeppelin screenshots, for instance.
> I can see the use case for solr-bin add-ons, kind of like how kubectl does plugins.
I tried making [making a proper Solr plugin out of this](https://github.com/gerlowskija/binsolr-zeppelin) a month or so back, but Solr's plugin framework isn't ready for this use case yet. (Currently, the plugin infrastructure lives in CoreContainer so only running Solr nodes have access to plugin jars. I was able to get my plugin installed, but SolrCLI couldn't see it to run ZeppelinTool).
That's all to say - I'm only going the direct bin/solr route here because other attempts at a plugin setup failed.
Can you expand on the kubectl-plugin-like approach you have in mind? I'm not against it necessarily, but I'm not sure I understand the benefits yet. The large majority of the zeppelin integration logic lives in Java, with `bin/solr` really just being a passthrough call into Java-land. What does moving that passthrough over to `bin/plugins/zeppelin.sh` really get us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-825355375
I think like Houston, I would prefer to see a bin/somethingelse instead of bin/solr on principle because this is not starting solr, this is starting/installing Zeppelin. On this point I'm "-0". Disclaimer I very cursory looked at the diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller edited a comment on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
markrmiller edited a comment on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-917359003
This one seems a bit odd to me as well. Given that Zeppelin is a server type app that is geared towards interfacing with a wide variety of data sources, even within a single notebook, it feels a bit odd to manage it via something out of Solr.
It's also fairly simple to setup and run - kind of feels more like a doc page. It's been a little while, but I believe you can even install the Solr interpreter as a helium plug-in with a few clicks in the UI.
Given it's difficult to test and play and anyone serious would likely just run zeppelin like you normally would, it almost seems simpler to just doc something like (outdated versions):
docker run -p 8080:8080 -d --name zeppelin apache/zeppelin:0.8.1
$ docker exec zeppelin ./bin/install-interpreter.sh --name solr --artifact com.lucidworks.zeppelin:zeppelin-solr:0.1.5
$ docker restart zeppelin # to load the interpreter
I don't even know you would have to restart these days, the helium stuff keeps improving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #74:
URL: https://github.com/apache/solr/pull/74#discussion_r697733633
##########
File path: solr/core/src/java/org/apache/solr/util/SolrCLI.java
##########
@@ -3689,6 +3697,329 @@ public AssertionFailureException(String message) {
}
}
+ public static class ZeppelinTool extends ToolBase {
+ private static final String ZEPP_VERSION = "0.10.0";
+ private static final String ZEPP_INSTALL_TYPE = "netinst";
+ private static final String ZEPP_UNPACK_DIR_NAME = "zeppelin-" + ZEPP_VERSION + "-bin-" + ZEPP_INSTALL_TYPE;
+ private static final String MIRROR_BASE = "http://apache.cs.utah.edu/zeppelin/zeppelin-" + ZEPP_VERSION + "/";
+ private static final String ZEPP_ARCHIVE = ZEPP_UNPACK_DIR_NAME + ".tgz";
+ private static final String ZEPP_ARCHIVE_URL = MIRROR_BASE + ZEPP_ARCHIVE;
+ private static final int PROCESS_WAIT_TIME_SECONDS = 90;
+
+ private static final String SOLR_INSTALL_DIR = System.getProperty("solr.install.dir");
+ private static final String ZEPP_INTERPRETER_TEMPLATE_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "server", "resources", "zeppelin-solr-interpreter.json.template").toAbsolutePath().toString();
+ private static final String ZEPP_INTERPRETER_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "server", "resources", "zeppelin-solr-interpreter.json").toAbsolutePath().toString();
+ private static final String ZEPP_BASE_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "zeppelin").toAbsolutePath().toString();
+ private static final String ZEPP_UNPACK_LOG_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, "untar-logs.txt").toAbsolutePath().toString();
+ private static final String ZEPP_ARCHIVE_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, ZEPP_ARCHIVE).toAbsolutePath().toString();
+ private static final String ZEPP_UNPACKED_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, ZEPP_UNPACK_DIR_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_DAEMON_EXE_NAME = "zeppelin-daemon.sh"; // Only valid on *nix
+ private static final String ZEPP_INTERP_EXE_NAME = "install-interpreter.sh"; // Only valid on *nix
+ private static final String ZEPP_DAEMON_ABS_PATH = Paths.get(ZEPP_UNPACKED_ABS_PATH, "bin", ZEPP_DAEMON_EXE_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_INTERP_ABS_PATH = Paths.get(ZEPP_UNPACKED_ABS_PATH, "bin", ZEPP_INTERP_EXE_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_SOLR_INTERPRETER_ID = "solr";
+
+ @Override
+ public String getName() { return "zeppelin"; }
+
+ @Override
+ public Option[] getOptions() {
+ return new Option[] {
+ Option.builder("z")
+ .desc("Only valid for action=update-interpreter. The Zeppelin URL to update. Defaults to http://localhost:8080")
+ .longOpt("zeppelinUrl")
+ .hasArg()
+ .argName("url")
+ .build(),
+ Option.builder("s")
+ .desc("The URL of a valid Solr instance. (e.g. http://localhost:7574/solr) Used to configure the" +
+ " Zeppelin interpreter. Defaults to values read from solr.in.sh")
+ .longOpt("solrUrl")
+ .hasArg()
+ .argName("url")
+ .required()
+ .build()
+ };
+ }
+
+ /*
+ * Since ZeppelinTool uses positional arguments not represented in getOptions, the help text generated from those
+ * values alone doesn't really cover the tool appropriately. This method returns custom help text, similar to that
+ * hardcoded in the solr/solr.cmd scripts themselves.
+ */
+ @Override
+ public void printHelpText() {
+ stdout.println("Usage: solr zeppelin bootstrap [--zeppelinUrl <url>] [--solrUrl <url>]");
+ stdout.println(" solr zeppelin clean");
+ stdout.println(" solr zeppelin start");
+ stdout.println(" solr zeppelin stop");
+ stdout.println(" solr zeppelin update_interpreter [--zeppelinUrl <url>] [--solrUrl <url>]");
+ stdout.println();
+ stdout.println(" Sets up and manages an Apache Zeppelin installation configured for use with a local Solr.");
+ stdout.println(" - 'bootstrap' downloads and starts Zeppelin in a 'zeppelin' folder within the Solr install");
+ stdout.println(" - 'clean' stops Zeppelin (if necessary) and removes the install created by 'bootstrap'.");
+ stdout.println(" - 'start' starts Zeppelin if not already running.");
+ stdout.println(" - 'stop' stops Zeppelin if not already stopped.");
+ stdout.println(" - 'update_interpreter' reinstalls the Zeppelin interpreter used to talk to Solr. Useful for");
+ stdout.println(" pointing Zeppelin to a different Solr cluster or changing other settings.");
+ stdout.println();
+ stdout.println(" This tool relies on management scripts that ship with Zeppelin - several of which are not");
+ stdout.println(" available on Windows. As a result this tool isn't supported in Windows environments at this time");
+ stdout.println();
+
+ // Append option information to help text
+ final HelpFormatter helpFormatter = new HelpFormatter();
+ try (final PrintWriter writer = new PrintWriter(stdout, true, StandardCharsets.UTF_8)) {
+ helpFormatter.printOptions(writer, 100, getToolOptions(this), helpFormatter.getLeftPadding(),
+ helpFormatter.getDescPadding());
+ }
+ }
+
+ enum Action {
+ bootstrap, clean, start, stop, update_interpreter
+ }
+
+ private boolean isValidAction(String userAction) {
+ try {
+ return Action.valueOf(userAction) != null;
+ } catch (IllegalArgumentException e) { // Thrown when the user-specified action is unrepresented in enum.
+ return false;
+ }
+ }
+
+ @Override
+ protected void runImpl(CommandLine cli) throws Exception {
+ final String zeppelinUrl = cli.getOptionValue("zeppelinUrl", "http://localhost:8080");
+ final String solrUrl = cli.getOptionValue("solrUrl");
+ final String[] positionalArgs = cli.getArgs();
+
+ // Validate provided action
+ final String validActions = Arrays.stream(Action.values()).map(x -> x.toString()).collect(Collectors.joining(",", "[", "]"));
+ if (positionalArgs.length != 1) {
+ echo("Zeppelin action not provided. Valid values are " + validActions);
+ exit(1);
+ } else if ("help".equals(positionalArgs[0])) {
+ printHelpText();
+ exit(0);
+ } else if (! isValidAction(positionalArgs[0])) {
+ echo("Invalid zeppelin action [" + positionalArgs[0] + "] provided. Valid values are " + validActions);
+ exit(1);
+ }
+
+ final Action action = Action.valueOf(positionalArgs[0].toLowerCase(Locale.ROOT));
+ switch (action) {
+ case bootstrap:
+ bootstrapZeppelinInstallation(cli, zeppelinUrl, solrUrl);
+ break;
+ case clean:
+ cleanZeppelinInstallation(cli);
+ break;
+ case start:
+ startZeppelinInstall();
+ break;
+ case stop:
+ stopZeppelin(cli);
+ break;
+ case update_interpreter:
+ updateSolrInterpreter(cli, zeppelinUrl, solrUrl);
+ break;
+ default:
+ echo("Invalid action value [" + action + "]; unable to proceed");
+ exit(1);
+ }
+ }
+
+ private void bootstrapZeppelinInstallation(CommandLine cli, String zeppelinUrl, String solrUrl) throws Exception {
+ final File zeppelinBaseDirFile = new File(ZEPP_BASE_ABS_PATH);
+ if (! zeppelinBaseDirFile.exists()) {
+ final boolean result = zeppelinBaseDirFile.mkdir();
+ if (! result) {
+ echo("Unable to create base directory [" + ZEPP_BASE_ABS_PATH + "] for Zeppelin install; exiting.");
+ exit(1);
+ }
+ }
+ echoIfVerbose("Zeppelin base dir created successfully at " + ZEPP_BASE_ABS_PATH, cli);
+
+ final File zeppelinArchiveFile = new File(ZEPP_ARCHIVE_ABS_PATH);
+ if (! zeppelinArchiveFile.exists()) {
+ echo("Downloading zeppelin; this may take a few minutes");
+ FileUtils.copyURLToFile(new URL(ZEPP_ARCHIVE_URL), zeppelinArchiveFile);
+ }
+
+ final File unpackedZeppelinArchiveFile = new File(ZEPP_UNPACKED_ABS_PATH);
+ if (! unpackedZeppelinArchiveFile.exists()) {
+ final File unpackLogs = new File(ZEPP_UNPACK_LOG_ABS_PATH);
+ Process ps = new ProcessBuilder().command("tar", "-xvf", ZEPP_ARCHIVE_ABS_PATH)
+ .redirectError(unpackLogs)
+ .redirectOutput(unpackLogs)
+ .directory(zeppelinBaseDirFile)
+ .start();
+ final boolean completed = ps.waitFor(PROCESS_WAIT_TIME_SECONDS, TimeUnit.SECONDS);
+ if (! completed) {
+ echo("Zeppelin could not be unpacked within " + PROCESS_WAIT_TIME_SECONDS + "s; unable to install Zeppelin");
+ exit(1);
+ }
+
+ if (0 != ps.exitValue()) {
+ echo("Attempt to unpack Zeppelin archive failed with code " + ps.exitValue() + "; exiting");
+ exit(1);
+ } else {
+ echoIfVerbose("Zeppelin successfully downloaded and unpacked to " + zeppelinBaseDirFile.getAbsolutePath(), cli);
+ }
+
+ if (unpackLogs.exists()) {
+ unpackLogs.delete();
+ }
+ }
+
+ echoIfVerbose("Finished initializing Zeppelin sandbox, attempting to start zeppelin...", cli);
+ startZeppelinInstall();
+ echoIfVerbose("Finished starting zeppelin, attempting to update zeppelin-solr plugin", cli);
+ runCommand(ZEPP_INTERP_ABS_PATH, "--name", "solr", "--artifact", "com.lucidworks.zeppelin:zeppelin-solr:0.1.6");
+ echoIfVerbose("Finished installing zeppelin-solr plugin, attempting to restart zepplin", cli);
+ runCommand(ZEPP_DAEMON_ABS_PATH, "restart");
+ echoIfVerbose("Finished restarting zeppelin, updating 'solr' interpreter to point to Solr URL: " + solrUrl, cli);
+ updateSolrInterpreter(cli, zeppelinUrl, solrUrl);
+ }
+
+ private void cleanZeppelinInstallation(CommandLine cli) throws Exception {
+ if (new File(ZEPP_DAEMON_ABS_PATH).exists() && isZeppelinRunning()) {
+ echoIfVerbose("Stopping zeppelin prior to 'clean'", cli);
+ stopZeppelin(cli);
+ }
+
+ FileUtils.deleteDirectory(new File(ZEPP_BASE_ABS_PATH));
+ }
+
+ private void startZeppelinInstall() throws Exception {
+ runCommand(ZEPP_DAEMON_ABS_PATH, "start");
+ }
+
+ private void stopZeppelin(CommandLine cli) throws Exception {
+ if (! new File(ZEPP_DAEMON_ABS_PATH).exists()) {
+ echoIfVerbose("No Zeppelin sandbox exists to stop; done.", cli);
+ exit(0);
+ }
+
+ if (! isZeppelinRunning()) {
+ echoIfVerbose("Zeppelin was already stopped", cli);
+ exit(0);
+ }
+
+ echoIfVerbose("Stopping Zeppelin using executable: " + ZEPP_DAEMON_ABS_PATH, cli);
+ runCommand(ZEPP_DAEMON_ABS_PATH, "stop");
+ }
+
+ private void ensureZeppelinAvailableAtUrl(CommandLine cli, String zeppelinUrl) throws Exception {
+ final HttpGet request = new HttpGet(new URI(zeppelinUrl + "/api/interpreter"));
+ final int maxAttempts = 30;
+ boolean success = false;
+ String lastErrorMessage = null;
+
+
+ try (final CloseableHttpClient httpClient = HttpClientUtil.createClient(new ModifiableSolrParams())) {
+ for (int i = 0; i < maxAttempts; i++) {
+ try {
+ final HttpResponse response = httpClient.execute(request);
+ if (response.getStatusLine().getStatusCode() <= 299) {
+ success = true;
+ break;
+ } else {
+ final ByteArrayOutputStream entityStream = new ByteArrayOutputStream();
+ response.getEntity().writeTo(entityStream);
+ lastErrorMessage = new String(entityStream.toByteArray(), StandardCharsets.UTF_8);
+ }
+ } catch (ConnectException e) {
+ lastErrorMessage = e.getMessage();
+ }
+ echoIfVerbose("Failed to reach zeppelin at url [" + zeppelinUrl + "]; error was: " + lastErrorMessage + ", retrying.", cli);
+ Thread.sleep(1000);
+ }
+
+ if (! success) {
+ echo("Failed to find Zeppelin at URL [" + zeppelinUrl + "]. Last error was: " + lastErrorMessage);
+ exit(-1);
+ }
+ }
+ }
+
+ private void updateSolrInterpreter(CommandLine cli, String zeppelinUrl, String solrUrl) throws Exception {
+ final File interpreterEntityFile = createInterpreterFromTemplate(solrUrl);
+ echoIfVerbose("Updating interpreter to have solrUrl: " + solrUrl, cli);
+ ensureZeppelinAvailableAtUrl(cli, zeppelinUrl);
+ // Zeppelin-solr creates the 'solr' interpreter setting upon install, so it always exists for a 'PUT' here.
+ try (final CloseableHttpClient httpClient = HttpClientUtil.createClient(new ModifiableSolrParams())) {
+ final HttpPut request = new HttpPut();
+ final URI requestURI = new URI(zeppelinUrl + "/api/interpreter/setting/" + ZEPP_SOLR_INTERPRETER_ID);
+ request.setURI(requestURI);
+ request.setEntity(new FileEntity(interpreterEntityFile, ContentType.APPLICATION_JSON));
+ final HttpResponse response = httpClient.execute(request);
+ if (response.getStatusLine().getStatusCode() > 299) {
+ echo("Received [" + response.getStatusLine().getStatusCode() + "] status when creating Solr interpreter; aborting.");
+ response.getEntity().writeTo(System.err);
+ exit(1);
+ }
+ } catch (Exception e) {
+ echo("Error encountered creating/updating Solr interpreter " + e.getMessage());
+ e.printStackTrace(System.err);
+ exit(1);
+ }
+
+ interpreterEntityFile.delete();
+ echoIfVerbose("Successfully created Solr interpreter", cli);
+ }
+
+ private File createInterpreterFromTemplate(String solrUrl) throws IOException {
+ final File interpreterTemplateFile = new File(ZEPP_INTERPRETER_TEMPLATE_ABS_PATH);
+ final String interpreterTemplate = FileUtils.readFileToString(interpreterTemplateFile, StandardCharsets.UTF_8);
+ final String interpreter = interpreterTemplate.replaceAll("@@SOLR_URL@@", solrUrl);
+
+ final File interpreterFile = new File(ZEPP_INTERPRETER_ABS_PATH);
+ FileUtils.writeStringToFile(interpreterFile, interpreter, StandardCharsets.UTF_8, false);
+ return interpreterFile;
+ }
+
+ private boolean isZeppelinRunning() throws Exception {
+ final int cmdStatus = runCommandForStatus(ZEPP_DAEMON_ABS_PATH, "restart");
+ return cmdStatus == 0;
+ }
+
+ private int runCommandForStatus(String executable, String... args) throws Exception {
+ final List<String> allArgs = new ArrayList<>();
+ allArgs.add(executable);
+ for (String arg : args) allArgs.add(arg);
+
+ final Process ps = new ProcessBuilder().command(allArgs).start();
Review comment:
*COMMAND_INJECTION:* This usage of java/lang/ProcessBuilder.command(Ljava/util/List;)Ljava/lang/ProcessBuilder; can be vulnerable to Command Injection [(details)](https://find-sec-bugs.github.io/bugs.htm#COMMAND_INJECTION)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/util/SolrCLI.java
##########
@@ -3689,6 +3697,329 @@ public AssertionFailureException(String message) {
}
}
+ public static class ZeppelinTool extends ToolBase {
+ private static final String ZEPP_VERSION = "0.10.0";
+ private static final String ZEPP_INSTALL_TYPE = "netinst";
+ private static final String ZEPP_UNPACK_DIR_NAME = "zeppelin-" + ZEPP_VERSION + "-bin-" + ZEPP_INSTALL_TYPE;
+ private static final String MIRROR_BASE = "http://apache.cs.utah.edu/zeppelin/zeppelin-" + ZEPP_VERSION + "/";
+ private static final String ZEPP_ARCHIVE = ZEPP_UNPACK_DIR_NAME + ".tgz";
+ private static final String ZEPP_ARCHIVE_URL = MIRROR_BASE + ZEPP_ARCHIVE;
+ private static final int PROCESS_WAIT_TIME_SECONDS = 90;
+
+ private static final String SOLR_INSTALL_DIR = System.getProperty("solr.install.dir");
+ private static final String ZEPP_INTERPRETER_TEMPLATE_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "server", "resources", "zeppelin-solr-interpreter.json.template").toAbsolutePath().toString();
+ private static final String ZEPP_INTERPRETER_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "server", "resources", "zeppelin-solr-interpreter.json").toAbsolutePath().toString();
+ private static final String ZEPP_BASE_ABS_PATH = Paths.get(SOLR_INSTALL_DIR, "zeppelin").toAbsolutePath().toString();
+ private static final String ZEPP_UNPACK_LOG_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, "untar-logs.txt").toAbsolutePath().toString();
+ private static final String ZEPP_ARCHIVE_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, ZEPP_ARCHIVE).toAbsolutePath().toString();
+ private static final String ZEPP_UNPACKED_ABS_PATH = Paths.get(ZEPP_BASE_ABS_PATH, ZEPP_UNPACK_DIR_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_DAEMON_EXE_NAME = "zeppelin-daemon.sh"; // Only valid on *nix
+ private static final String ZEPP_INTERP_EXE_NAME = "install-interpreter.sh"; // Only valid on *nix
+ private static final String ZEPP_DAEMON_ABS_PATH = Paths.get(ZEPP_UNPACKED_ABS_PATH, "bin", ZEPP_DAEMON_EXE_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_INTERP_ABS_PATH = Paths.get(ZEPP_UNPACKED_ABS_PATH, "bin", ZEPP_INTERP_EXE_NAME).toAbsolutePath().toString();
+ private static final String ZEPP_SOLR_INTERPRETER_ID = "solr";
+
+ @Override
+ public String getName() { return "zeppelin"; }
+
+ @Override
+ public Option[] getOptions() {
+ return new Option[] {
+ Option.builder("z")
+ .desc("Only valid for action=update-interpreter. The Zeppelin URL to update. Defaults to http://localhost:8080")
+ .longOpt("zeppelinUrl")
+ .hasArg()
+ .argName("url")
+ .build(),
+ Option.builder("s")
+ .desc("The URL of a valid Solr instance. (e.g. http://localhost:7574/solr) Used to configure the" +
+ " Zeppelin interpreter. Defaults to values read from solr.in.sh")
+ .longOpt("solrUrl")
+ .hasArg()
+ .argName("url")
+ .required()
+ .build()
+ };
+ }
+
+ /*
+ * Since ZeppelinTool uses positional arguments not represented in getOptions, the help text generated from those
+ * values alone doesn't really cover the tool appropriately. This method returns custom help text, similar to that
+ * hardcoded in the solr/solr.cmd scripts themselves.
+ */
+ @Override
+ public void printHelpText() {
+ stdout.println("Usage: solr zeppelin bootstrap [--zeppelinUrl <url>] [--solrUrl <url>]");
+ stdout.println(" solr zeppelin clean");
+ stdout.println(" solr zeppelin start");
+ stdout.println(" solr zeppelin stop");
+ stdout.println(" solr zeppelin update_interpreter [--zeppelinUrl <url>] [--solrUrl <url>]");
+ stdout.println();
+ stdout.println(" Sets up and manages an Apache Zeppelin installation configured for use with a local Solr.");
+ stdout.println(" - 'bootstrap' downloads and starts Zeppelin in a 'zeppelin' folder within the Solr install");
+ stdout.println(" - 'clean' stops Zeppelin (if necessary) and removes the install created by 'bootstrap'.");
+ stdout.println(" - 'start' starts Zeppelin if not already running.");
+ stdout.println(" - 'stop' stops Zeppelin if not already stopped.");
+ stdout.println(" - 'update_interpreter' reinstalls the Zeppelin interpreter used to talk to Solr. Useful for");
+ stdout.println(" pointing Zeppelin to a different Solr cluster or changing other settings.");
+ stdout.println();
+ stdout.println(" This tool relies on management scripts that ship with Zeppelin - several of which are not");
+ stdout.println(" available on Windows. As a result this tool isn't supported in Windows environments at this time");
+ stdout.println();
+
+ // Append option information to help text
+ final HelpFormatter helpFormatter = new HelpFormatter();
+ try (final PrintWriter writer = new PrintWriter(stdout, true, StandardCharsets.UTF_8)) {
+ helpFormatter.printOptions(writer, 100, getToolOptions(this), helpFormatter.getLeftPadding(),
+ helpFormatter.getDescPadding());
+ }
+ }
+
+ enum Action {
+ bootstrap, clean, start, stop, update_interpreter
+ }
+
+ private boolean isValidAction(String userAction) {
+ try {
+ return Action.valueOf(userAction) != null;
+ } catch (IllegalArgumentException e) { // Thrown when the user-specified action is unrepresented in enum.
+ return false;
+ }
+ }
+
+ @Override
+ protected void runImpl(CommandLine cli) throws Exception {
+ final String zeppelinUrl = cli.getOptionValue("zeppelinUrl", "http://localhost:8080");
+ final String solrUrl = cli.getOptionValue("solrUrl");
+ final String[] positionalArgs = cli.getArgs();
+
+ // Validate provided action
+ final String validActions = Arrays.stream(Action.values()).map(x -> x.toString()).collect(Collectors.joining(",", "[", "]"));
+ if (positionalArgs.length != 1) {
+ echo("Zeppelin action not provided. Valid values are " + validActions);
+ exit(1);
+ } else if ("help".equals(positionalArgs[0])) {
+ printHelpText();
+ exit(0);
+ } else if (! isValidAction(positionalArgs[0])) {
+ echo("Invalid zeppelin action [" + positionalArgs[0] + "] provided. Valid values are " + validActions);
+ exit(1);
+ }
+
+ final Action action = Action.valueOf(positionalArgs[0].toLowerCase(Locale.ROOT));
+ switch (action) {
+ case bootstrap:
+ bootstrapZeppelinInstallation(cli, zeppelinUrl, solrUrl);
+ break;
+ case clean:
+ cleanZeppelinInstallation(cli);
+ break;
+ case start:
+ startZeppelinInstall();
+ break;
+ case stop:
+ stopZeppelin(cli);
+ break;
+ case update_interpreter:
+ updateSolrInterpreter(cli, zeppelinUrl, solrUrl);
+ break;
+ default:
+ echo("Invalid action value [" + action + "]; unable to proceed");
+ exit(1);
+ }
+ }
+
+ private void bootstrapZeppelinInstallation(CommandLine cli, String zeppelinUrl, String solrUrl) throws Exception {
+ final File zeppelinBaseDirFile = new File(ZEPP_BASE_ABS_PATH);
+ if (! zeppelinBaseDirFile.exists()) {
+ final boolean result = zeppelinBaseDirFile.mkdir();
+ if (! result) {
+ echo("Unable to create base directory [" + ZEPP_BASE_ABS_PATH + "] for Zeppelin install; exiting.");
+ exit(1);
+ }
+ }
+ echoIfVerbose("Zeppelin base dir created successfully at " + ZEPP_BASE_ABS_PATH, cli);
+
+ final File zeppelinArchiveFile = new File(ZEPP_ARCHIVE_ABS_PATH);
+ if (! zeppelinArchiveFile.exists()) {
+ echo("Downloading zeppelin; this may take a few minutes");
+ FileUtils.copyURLToFile(new URL(ZEPP_ARCHIVE_URL), zeppelinArchiveFile);
+ }
+
+ final File unpackedZeppelinArchiveFile = new File(ZEPP_UNPACKED_ABS_PATH);
+ if (! unpackedZeppelinArchiveFile.exists()) {
+ final File unpackLogs = new File(ZEPP_UNPACK_LOG_ABS_PATH);
+ Process ps = new ProcessBuilder().command("tar", "-xvf", ZEPP_ARCHIVE_ABS_PATH)
Review comment:
*COMMAND_INJECTION:* This usage of java/lang/ProcessBuilder.command([Ljava/lang/String;)Ljava/lang/ProcessBuilder; can be vulnerable to Command Injection [(details)](https://find-sec-bugs.github.io/bugs.htm#COMMAND_INJECTION)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija edited a comment on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
gerlowskija edited a comment on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-819722432
Hey @HoustonPutman Thanks for chiming in. A few replies inline.
> what does this provide over running Zeppelin locally yourself?
It automates all of the needed setup behind a single command: `bin/solr zeppelin bootstrap`. So in a word: convenience. You're right that Solr doesn't "know" about Zeppelin the same way that it knows about ZK, though Zeppelin isn't a totally random tool either. Our ["Math Expressions" docs](https://solr.apache.org/guide/8_8/math-expressions.html) make heavy use of Zeppelin screenshots, for instance.
> I can see the use case for solr-bin add-ons, kind of like how kubectl does plugins.
I tried making [making a proper Solr plugin out of this](https://github.com/gerlowskija/binsolr-zeppelin) a month or so back, but Solr's plugin framework isn't ready for this use case yet. (Currently, the plugin infrastructure lives in CoreContainer so only running Solr nodes have access to plugin jars. I was able to get my plugin installed, but SolrCLI couldn't see it to run ZeppelinTool).
That's all to say - I'm only going the direct bin/solr route here because other attempts at a plugin setup failed.
Can you expand on the kubectl-plugin-like approach you have in mind? I'm not against it necessarily, but I'm not sure I understand the benefits yet. The large majority of the zeppelin integration logic lives in Java, with `bin/solr` really just being a passthrough call into Java-land. What does moving that passthrough over to `bin/plugins/zeppelin.sh` really get us? i.e. if the Java logic stays where it is then there's no real separation. If the Java logic moves then `bin/solr` will need to find the additional jar, modify the classpath it launches the JVM with, etc....all of which sounds dangerously like yet-another-Solr-plugin-framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-818962583
I can see the use case for solr-bin add-ons, kind of like how `kubectl` does [plugins](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/). Would it be possible to refactor this into a separate file (maybe a bin/plugins/zepelin.sh or something), then enable it through `/bin/solr plugin zepelin`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
epugh commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-819009061
> I can see the use case for solr-bin add-ons, kind of like how `kubectl` does [plugins](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/). Would it be possible to refactor this into a separate file (maybe a bin/plugins/zepelin.sh or something), then enable it through `/bin/solr plugin zepelin`?
Interesting idea! CAuse eventually you do `bin/solr package zeppelin` and then `bin/solr plugin zeppelin blah blah blah`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller edited a comment on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
markrmiller edited a comment on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-917359003
This one seems a bit odd to me as well. Given that Zeppelin is a server type app that is geared towards interfacing with a wide variety of data sources, even within a single notebook, it feels a bit odd to manage it via something out of Solr.
It's also fairly simple to setup and run - kind of feels more like a doc page. It's been a little while, but I believe you can even install the Solr interpreter as a helium plug-in with a few clicks in the UI.
Given it's difficult to test and mainly for test & play and anyone serious would likely just run zeppelin like you normally would, it almost seems simpler to just doc something like (outdated versions):
docker run -p 8080:8080 -d --name zeppelin apache/zeppelin:0.8.1
$ docker exec zeppelin ./bin/install-interpreter.sh --name solr --artifact com.lucidworks.zeppelin:zeppelin-solr:0.1.5
$ docker restart zeppelin # to load the interpreter
I don't even know you would have to restart these days, the helium stuff keeps improving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-818020414
@gerlowskija, what does this provide over running Zeppelin locally yourself? I think `bin/solr` having an integration with Zookeeper makes a lot of sense, because Zookeeper is so tightly integrated inside of Solr. However Solr doesn't know what Zepplin is (correct me if I'm wrong here). I think it would be awesome to provide a detailed guide on using Solr and Zeppelin together, and either point to documentation on running Zeppelin, or have a short tutorial in the ref guide. However I'm not sure it's appropriate to have a command to start an unrelated service within `bin/solr`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
epugh commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-818236266
What was so interesting about this was that it seemed to lower the barrier to trying something new out, i.e using Zeppelin. There is a lot of power around analytics locked up in Solr, and it's kind of hard to access. I've mulled over trying to build out more analytics capablities in the Solr Admin, but that is a very steep climb. I also thought about Jupyter, but the work that went into the analytics documentation was all based on Zeppelin, (plus another Apache project!) so it made sense to use it.
`bin/solr` to me feels like an entry to a whole grab bag of tools, not just core "stop/start the Solr service".
Maybe we should have had many other scripts in in `bin/` directory?
`bin/solr zeppelin` or maybe `bin/solr demo-analytics` is all a lot easier than downloading and installing Zeppelin, and getting it configured to talk to Solr.
I do think documenting how to connect Solr and Zeppelin would be a good thing! Maybe we add it to the tutorial?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
markrmiller commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-917359003
This one seems a bit odd ti me as well. Given that Zeppelin is a server type app that is geared towards interfacing with a wide variety of data sources, even within a single notebook, it feels a bit odd to manage it via something out of Solr.
It's also fairly simple to setup and run - kind of feels more like a doc page. It's been a little while, but I believe you can even install the Solr interpreter as a helium plug-in with a few clicks in the UI.
Given it's difficult to test and play and anyone serious would likely just run zeppelin like you normally would, it almost seems simpler to just doc something like (outdated versions):
docker run -p 8080:8080 -d --name zeppelin apache/zeppelin:0.8.1
$ docker exec zeppelin ./bin/install-interpreter.sh --name solr --artifact com.lucidworks.zeppelin:zeppelin-solr:0.1.5
$ docker restart zeppelin # to load the interpreter
I don't even know you would have to restart these days, the helium stuff keeps improving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on pull request #74:
URL: https://github.com/apache/solr/pull/74#issuecomment-822662346
Sorry, should have read through the PR more thoroughly. Agree that the passthrough isn't too bad if it's that small. There are a few additional questions I have but they are all related to having the Zepplin CLI code in a package (when that is supported in the future). So no need in asking them yet 🙂
Thanks for the thorough response. Is there a ticket tracking Plugins supporting CLI additions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a change in pull request #74: SOLR-15080: Add 'zeppelin' integration to bin/solr
Posted by GitBox <gi...@apache.org>.
epugh commented on a change in pull request #74:
URL: https://github.com/apache/solr/pull/74#discussion_r697560086
##########
File path: solr/core/src/java/org/apache/solr/util/SolrCLI.java
##########
@@ -411,24 +428,15 @@ else if ("package".equals(toolType))
}
private static void displayToolOptions() throws Exception {
- HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("healthcheck", getToolOptions(new HealthcheckTool()));
- formatter.printHelp("status", getToolOptions(new StatusTool()));
- formatter.printHelp("api", getToolOptions(new ApiTool()));
- formatter.printHelp("create_collection", getToolOptions(new CreateCollectionTool()));
- formatter.printHelp("create_core", getToolOptions(new CreateCoreTool()));
- formatter.printHelp("create", getToolOptions(new CreateTool()));
- formatter.printHelp("delete", getToolOptions(new DeleteTool()));
- formatter.printHelp("config", getToolOptions(new ConfigTool()));
- formatter.printHelp("run_example", getToolOptions(new RunExampleTool()));
- formatter.printHelp("upconfig", getToolOptions(new ConfigSetUploadTool()));
- formatter.printHelp("downconfig", getToolOptions(new ConfigSetDownloadTool()));
- formatter.printHelp("rm", getToolOptions(new ZkRmTool()));
- formatter.printHelp("cp", getToolOptions(new ZkCpTool()));
- formatter.printHelp("mv", getToolOptions(new ZkMvTool()));
- formatter.printHelp("ls", getToolOptions(new ZkLsTool()));
- formatter.printHelp("export", getToolOptions(new ExportTool()));
- formatter.printHelp("package", getToolOptions(new PackageTool()));
+ final HelpFormatter formatter = new HelpFormatter();
Review comment:
Should this be moved to a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org