You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "veghlaci05 (via GitHub)" <gi...@apache.org> on 2023/02/14 10:28:43 UTC
[GitHub] [hive] veghlaci05 opened a new pull request, #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
veghlaci05 opened a new pull request, #4060:
URL: https://github.com/apache/hive/pull/4060
### What changes were proposed in this pull request?
Introduce liquibase to handle the schema evolution under the hood, while leave Schematool API interface unchanged.
### Why are the changes needed?
To replace the custom schema evolution logic with a well established robust and industry standard one.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manuall and through unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1516558015
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [210 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1531814412
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1106991640
##########
standalone-metastore/metastore-server/pom.xml:
##########
@@ -25,6 +25,11 @@
<standalone.metastore.path.to.root>..</standalone.metastore.path.to.root>
</properties>
<dependencies>
+ <dependency>
+ <groupId>org.liquibase</groupId>
+ <artifactId>liquibase-core</artifactId>
+ <version>4.19.0</version>
Review Comment:
Since this is a crucial dependency, I believe its best to keep the version - `4.19.0` as a separate version tag in standalone-metastore pom file and reference the version tag here.
Something like this -
`<version>${liquibase.version}</version>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on a diff in pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1108297390
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/CommandBuilder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool;
+
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.FileReader;
+import java.io.IOException;
+
+public class CommandBuilder {
+
+ private static final Logger LOG = LoggerFactory.getLogger(MetastoreSchemaTool.class);
+
+ private static final String PASSWD_MASK = "[passwd stripped]";
+
+ protected final HiveSchemaHelper.MetaStoreConnectionInfo connectionInfo;
+
+ private boolean verbose = false;
+
+ public CommandBuilder(HiveSchemaHelper.MetaStoreConnectionInfo connectionInfo) {
+ this.connectionInfo = connectionInfo;
+ }
+
+ public CommandBuilder setVerbose(boolean verbose) {
+ this.verbose = verbose;
+ return this;
+ }
+
+ public HiveSchemaHelper.MetaStoreConnectionInfo getConnectionInfo() {
+ return connectionInfo;
+ }
+
+ public boolean isVerbose() {
+ return verbose;
+ }
+
+ public String[] buildToRun(String sqlScriptFile) {
+ return argsWith(connectionInfo.getPassword(), sqlScriptFile);
+ }
+
+ public String buildToLog(String sqlScriptFile) throws IOException {
+ if (verbose) {
+ logScript(sqlScriptFile);
+ }
+ return StringUtils.join(argsWith(PASSWD_MASK, sqlScriptFile), " ");
+ }
+
+ protected String[] argsWith(String password, String sqlScriptFile) {
+ return new String[]
+ {
+ "-u", connectionInfo.getUrl(),
+ "-d", connectionInfo.getDriver(),
+ "-n", connectionInfo.getUsername(),
+ "-p", password,
+ "--isolation=TRANSACTION_READ_COMMITTED",
+ "-f", sqlScriptFile
+ };
+ }
+
+ private void logScript(String sqlScriptFile) throws IOException {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Going to invoke file that contains:");
Review Comment:
Can we log something like "Invoking {sqScriptFile} containing:"
This will help verify the filePath as well while Logging the queries. WDYT
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/RootTask.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.MetaStoreSchemaInfoFactory;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.TaskContext;
+
+import java.io.IOException;
+import java.util.Set;
+
+class RootTask extends SchemaToolTask {
+
+ private static final Set<String> VALID_DB_TYPES = ImmutableSet.of(HiveSchemaHelper.DB_DERBY,
+ HiveSchemaHelper.DB_HIVE, HiveSchemaHelper.DB_MSSQL, HiveSchemaHelper.DB_MYSQL,
+ HiveSchemaHelper.DB_POSTGRACE, HiveSchemaHelper.DB_ORACLE);
Review Comment:
nit: Typo in DB_POSTGRES
I know this is an old constant, but while we are at it we can fix this. It doesnt have many usage, only 3 including 2 in this file.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SqlLineScriptExecutor.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.hms.ScriptExecutor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import sqlline.SqlLine;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.PrintStream;
+
+public class SqlLineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SqlLineScriptExecutor.class);
+
+ private final CommandBuilder builder;
+
+ @Override
+ public void execSql(String scriptPath) throws IOException {
+ // run the script using SqlLine
+ SqlLine sqlLine = new SqlLine();
+ ByteArrayOutputStream outputForLog = null;
+ if (!builder.isVerbose()) {
+ OutputStream out;
+ if (LOG.isDebugEnabled()) {
+ out = outputForLog = new ByteArrayOutputStream();
+ } else {
+ out = new NullOutputStream();
+ }
+ sqlLine.setOutputStream(new PrintStream(out));
+ System.setProperty("sqlline.silent", "true");
+ }
+ LOG.info("Going to run command <" + builder.buildToLog(scriptPath) + ">");
+ SqlLine.Status status = sqlLine.begin(builder.buildToRun(scriptPath), null, false);
+ if (LOG.isDebugEnabled() && outputForLog != null) {
+ LOG.debug("Received following output from Sqlline:");
+ LOG.debug(outputForLog.toString("UTF-8"));
Review Comment:
These 2 can be merged into one LOG.Debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1218019110
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveUpdateTask.java:
##########
Review Comment:
The benefit would be quite small (slightly different error message on the console), but it would require a medium-sized change as in the current infrastructure the task is not knowing to which command(s) it is mapped to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1106991640
##########
standalone-metastore/metastore-server/pom.xml:
##########
@@ -25,6 +25,11 @@
<standalone.metastore.path.to.root>..</standalone.metastore.path.to.root>
</properties>
<dependencies>
+ <dependency>
+ <groupId>org.liquibase</groupId>
+ <artifactId>liquibase-core</artifactId>
+ <version>4.19.0</version>
Review Comment:
Since this is a major dependency, I believe its best to keep the version - `4.19.0` as a separate version tag in standalone-metastore pom file and reference the version tag here.
Something like this -
`<version>${liquibase.version}</version>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1514884590
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [210 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kokila-19 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1214407305
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/task/SchemaToolTaskFactory.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.task;
+
+import org.apache.commons.cli.Option;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolCommandLine;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParserFactory;
+
+import java.util.Set;
+
+/**
+ * Responsible for creating the {@link SchemaToolTask} chain based on the input provided via the {@link SchemaToolCommandLine}
+ * object
+ */
+public class SchemaToolTaskFactory {
+
+ private final SchemaToolTaskProvider[] providers;
+ private final NestedScriptParserFactory scriptParserFactory;
+
+ /**
+ * Creates the {@link SchemaToolTask} chain based on the input provided via the {@link SchemaToolCommandLine}.
+ * Checks for dangling command line arguments (arguments which are not used by any of the tasks in the created task chain).
+ * @param cmdLine A {@link SchemaToolCommandLine} instance containing the parsed command line arguments.
+ * @return Returns with the created {@link SchemaToolTask} chain.
+ * @throws HiveMetaException Thrown in the following cases:
+ * <ul>
+ * <li>Dangling arguments found</li>
+ * <li>No task found for the given command line arguments</li>
+ * <li>None of the available {@link SchemaToolTaskProvider} instances are supporting the database type provided via the
+ * '-dbType' option. For list of supported DB types see: {@link org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper} constants</li>
+ * </ul>
+ */
+ public SchemaToolTask getTask(SchemaToolCommandLine cmdLine) throws HiveMetaException {
+ for(SchemaToolTaskProvider provider : providers) {
+ if(!provider.getSupportedDatabases().contains(cmdLine.getDbType().toLowerCase())) {
+ continue;
+ }
+ SchemaToolTask rootTask = new RootTask(scriptParserFactory);
+ for (Option command : cmdLine.getOptions()) {
+ SchemaToolTask task = provider.getTask(command.getOpt());
+ if (task != null) {
+ rootTask.addChild(task);
+ //Check for unused command line arguments
+ Set<String> arguments = rootTask.getUsedCommandLineArguments();
+ arguments.add(command.getOpt());
+ arguments = cmdLine.getDanglingArguments(arguments);
+ if (arguments.size() > 0) {
+ throw new HiveMetaException("The following arguments are not used by the given command (" + command + "): " + String.join(",", arguments));
Review Comment:
Hi laszlo, i am curious to know is it really required to throw exception here. Can a warning be thrown and then continue with executing schema tool? will there be any impact ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1704447829
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27032: Introduce liquibase for HMS schema evolution [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1862682182
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[208 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&inNewCodePeriod=true)
[6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221446510
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveSchemaInfo.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+class HiveSchemaInfo extends SchemaInfo {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveSchemaInfo.class);
+ private static final String UPGRADE_FILE_PREFIX = "upgrade-";
+ protected static final String VERSION_UPGRADE_LIST = "upgrade.order";
+ protected static final String INITIAL_VERSION = "0.0.0";
+
+ String[] hiveSchemaVersions = null;
+
+ @Override
+ public List<String> getUnappliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ String schemaVersion = INITIAL_VERSION;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ } else throw e;
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+
+ // check if we are already at the required schema level
+ if (isVersionCompatible(getRequiredHiveSchemaVersion(), schemaVersion)) {
+ return upgradeScriptList;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ int firstScript = -1;
+ for (int i=0; i < hiveSchemaVersions.length; i++) {
+ if (hiveSchemaVersions[i].startsWith(schemaVersion + "-to-")) {
+ firstScript = i;
+ }
+ }
+ if (firstScript == -1) {
+ throw new HiveMetaException("Unknown version specified for upgrade " + schemaVersion + " Metastore schema may be too old or newer");
Review Comment:
Good point, this is no longer valid. Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1535452128
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [204 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1512160008
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [206 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1511805233
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [206 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1502487017
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [204 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1214113485
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/SchemaInfo.java:
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+
+import java.io.File;
+import java.util.List;
+
+/**
+ * Provides information about the HMS or Hive schema
+ */
+public abstract class SchemaInfo {
+
+ /**
+ * This must be updated accordingly in case of breaking schema changes.
+ * For example: adding a new column to an HMS table.
+ */
+ private static final String MIN_HMS_SCHEMA_VERSION = "4.0.0";
Review Comment:
No, these are hardcoded values and representing the minimum required versions by the application itself. This means these are strictly bound to the current application code, and can be changed only by developers when they implement something which also changes the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1222912321
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
Review Comment:
`BufferedWriter` is a wrapper over `Writer` implementations. It simply adds buffering, and delegates all calls to the underlying `Writer`, including the `close()` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
URL: https://github.com/apache/hive/pull/4060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27032: Introduce liquibase for HMS schema evolution [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1861664875
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[215 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&inNewCodePeriod=true)
[6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aturoczy commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221191257
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
Review Comment:
Why it needs to delete?
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));
+ beeLine.getOpts().setSilent(true);
+ }
+ beeLine.getOpts().setAllowMultiLineCommand(false);
+ beeLine.getOpts().setIsolation(TRANSACTION_READ_COMMITTED);
+ // We can be pretty sure that an entire line can be processed as a single command since
+ // we always add a line separator at the end while calling dbCommandParser.buildCommand.
+ beeLine.getOpts().setEntireLineAsCommand(true);
+ LOG.debug("Going to run command <" + commandBuilder.buildToLog(sqlScriptFile) + ">");
+ int status = beeLine.begin(commandBuilder.buildToRun(sqlScriptFile), null, false);
+ if (status != 0) {
+ throw new IOException("Schema script failed, errorcode " + status);
Review Comment:
I would change it to error code:
I know, I know I found the biggest thing in the history :D
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));
+ beeLine.getOpts().setSilent(true);
+ }
+ beeLine.getOpts().setAllowMultiLineCommand(false);
+ beeLine.getOpts().setIsolation(TRANSACTION_READ_COMMITTED);
+ // We can be pretty sure that an entire line can be processed as a single command since
+ // we always add a line separator at the end while calling dbCommandParser.buildCommand.
+ beeLine.getOpts().setEntireLineAsCommand(true);
+ LOG.debug("Going to run command <" + commandBuilder.buildToLog(sqlScriptFile) + ">");
+ int status = beeLine.begin(commandBuilder.buildToRun(sqlScriptFile), null, false);
+ if (status != 0) {
+ throw new IOException("Schema script failed, errorcode " + status);
+ }
+ }
+ }
+
+ public BeelineScriptExecutor(NestedScriptParser dbCommandParser, CommandBuilder commandBuilder) {
Review Comment:
Constructor is on bottom?! https://www.youtube.com/watch?v=i1ojUmdF42U
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveSchemaInfo.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+class HiveSchemaInfo extends SchemaInfo {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveSchemaInfo.class);
+ private static final String UPGRADE_FILE_PREFIX = "upgrade-";
+ protected static final String VERSION_UPGRADE_LIST = "upgrade.order";
+ protected static final String INITIAL_VERSION = "0.0.0";
+
+ String[] hiveSchemaVersions = null;
+
+ @Override
+ public List<String> getUnappliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ String schemaVersion = INITIAL_VERSION;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ } else throw e;
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+
+ // check if we are already at the required schema level
+ if (isVersionCompatible(getRequiredHiveSchemaVersion(), schemaVersion)) {
+ return upgradeScriptList;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ int firstScript = -1;
+ for (int i=0; i < hiveSchemaVersions.length; i++) {
+ if (hiveSchemaVersions[i].startsWith(schemaVersion + "-to-")) {
+ firstScript = i;
+ }
+ }
+ if (firstScript == -1) {
+ throw new HiveMetaException("Unknown version specified for upgrade " + schemaVersion + " Metastore schema may be too old or newer");
Review Comment:
Newer is also possible?
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveSchemaInfo.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+class HiveSchemaInfo extends SchemaInfo {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveSchemaInfo.class);
+ private static final String UPGRADE_FILE_PREFIX = "upgrade-";
+ protected static final String VERSION_UPGRADE_LIST = "upgrade.order";
+ protected static final String INITIAL_VERSION = "0.0.0";
+
+ String[] hiveSchemaVersions = null;
+
+ @Override
+ public List<String> getUnappliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ String schemaVersion = INITIAL_VERSION;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ } else throw e;
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+
+ // check if we are already at the required schema level
+ if (isVersionCompatible(getRequiredHiveSchemaVersion(), schemaVersion)) {
+ return upgradeScriptList;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ int firstScript = -1;
+ for (int i=0; i < hiveSchemaVersions.length; i++) {
+ if (hiveSchemaVersions[i].startsWith(schemaVersion + "-to-")) {
+ firstScript = i;
+ }
+ }
+ if (firstScript == -1) {
+ throw new HiveMetaException("Unknown version specified for upgrade " + schemaVersion + " Metastore schema may be too old or newer");
+ }
+
+ for (int i=firstScript; i < hiveSchemaVersions.length; i++) {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersions[i]);
+ upgradeScriptList.add(scriptFile);
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public List<String> getAppliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+ String schemaVersion;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ return upgradeScriptList;
+ } else throw e;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ for (String hiveSchemaVersion : hiveSchemaVersions) {
+ if (hiveSchemaVersion.startsWith(schemaVersion + "-to-")) {
+ break;
+ } else {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersion);
+ upgradeScriptList.add(scriptFile);
+ }
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public String getSchemaVersion() throws HiveMetaException {
+ try (Connection metastoreDbConnection = HiveSchemaHelper.getConnectionToMetastore(connectionInfo, conf, "SYS");
+ Statement stmt = metastoreDbConnection.createStatement()) {
+ ResultSet res = stmt.executeQuery("select t.SCHEMA_VERSION from VERSION t");
+ if (!res.next()) {
+ throw new HiveMetaException("Could not find version info in Hive VERSION table.");
+ }
+ String currentSchemaVersion = res.getString(1);
+ if (res.next()) {
+ throw new HiveMetaException("Multiple versions were found in version table.");
+ }
+ return currentSchemaVersion;
+ } catch (SQLException e) {
+ throw new HiveMetaException("Failed to get schema version, Cause:" + e.getMessage());
+ }
+ }
+
+ @Override
+ public String getCreateUserScript() throws HiveMetaException {
+ String createScript = CREATE_USER_PREFIX + "." + connectionInfo + SQL_FILE_EXTENSION;
+ File scriptFile = new File(getMetaStoreScriptDir() + File.separatorChar + createScript);
+ // check if the file exists
+ if (!scriptFile.exists()) {
+ throw new HiveMetaException("Unable to find create user file, expected: " + scriptFile.getAbsolutePath());
+ }
+ return createScript;
+ }
+
+ // format the upgrade script name eg upgrade-x-y-dbType.sql
+ private String generateUpgradeFileName(String fileVersion) {
+ return UPGRADE_FILE_PREFIX + fileVersion + "." + connectionInfo.getDbType() + SQL_FILE_EXTENSION;
+ }
+
+ private void loadAllUpgradeScripts() throws HiveMetaException {
+ // load upgrade order for the given dbType
+ List<String> upgradeOrderList = new ArrayList<>();
+ String upgradeListFile = getMetaStoreScriptDir() + File.separator + VERSION_UPGRADE_LIST + "." + connectionInfo.getDbType();
+ try (FileReader fr = new FileReader(upgradeListFile);
+ BufferedReader bfReader = new BufferedReader(fr)) {
+ String currSchemaVersion;
+ while ((currSchemaVersion = bfReader.readLine()) != null) {
+ upgradeOrderList.add(currSchemaVersion.trim());
Review Comment:
+1 for trim :)
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveSchemaInfo.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+class HiveSchemaInfo extends SchemaInfo {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveSchemaInfo.class);
+ private static final String UPGRADE_FILE_PREFIX = "upgrade-";
+ protected static final String VERSION_UPGRADE_LIST = "upgrade.order";
+ protected static final String INITIAL_VERSION = "0.0.0";
+
+ String[] hiveSchemaVersions = null;
+
+ @Override
+ public List<String> getUnappliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ String schemaVersion = INITIAL_VERSION;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ } else throw e;
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+
+ // check if we are already at the required schema level
+ if (isVersionCompatible(getRequiredHiveSchemaVersion(), schemaVersion)) {
+ return upgradeScriptList;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ int firstScript = -1;
+ for (int i=0; i < hiveSchemaVersions.length; i++) {
+ if (hiveSchemaVersions[i].startsWith(schemaVersion + "-to-")) {
+ firstScript = i;
+ }
+ }
+ if (firstScript == -1) {
+ throw new HiveMetaException("Unknown version specified for upgrade " + schemaVersion + " Metastore schema may be too old or newer");
+ }
+
+ for (int i=firstScript; i < hiveSchemaVersions.length; i++) {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersions[i]);
+ upgradeScriptList.add(scriptFile);
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public List<String> getAppliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+ String schemaVersion;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ return upgradeScriptList;
+ } else throw e;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ for (String hiveSchemaVersion : hiveSchemaVersions) {
+ if (hiveSchemaVersion.startsWith(schemaVersion + "-to-")) {
+ break;
+ } else {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersion);
+ upgradeScriptList.add(scriptFile);
+ }
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public String getSchemaVersion() throws HiveMetaException {
+ try (Connection metastoreDbConnection = HiveSchemaHelper.getConnectionToMetastore(connectionInfo, conf, "SYS");
+ Statement stmt = metastoreDbConnection.createStatement()) {
+ ResultSet res = stmt.executeQuery("select t.SCHEMA_VERSION from VERSION t");
+ if (!res.next()) {
+ throw new HiveMetaException("Could not find version info in Hive VERSION table.");
+ }
+ String currentSchemaVersion = res.getString(1);
+ if (res.next()) {
+ throw new HiveMetaException("Multiple versions were found in version table.");
+ }
+ return currentSchemaVersion;
+ } catch (SQLException e) {
+ throw new HiveMetaException("Failed to get schema version, Cause:" + e.getMessage());
Review Comment:
Cause by: and missing whitespace
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveTaskProvider.java:
##########
Review Comment:
Considerable idea. :)
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
Review Comment:
As this is a stream should not be in a try block to close it? I do not see if it is closed at all.
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
Review Comment:
what happens if the user does not have permission to delete it? Should not be in a try catch?
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveUpdateTask.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolCommandLine;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.TaskContext;
+import org.apache.hive.beeline.schematool.HiveSchemaTool;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+
+class HiveUpdateTask extends SchemaToolTask {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveUpdateTask.class);
+
+ private static final String VERSION_UPGRADE_SCRIPT =
+ "USE SYS\n" +
+ "DROP TABLE IF EXISTS `VERSION`\n" +
+ "%s\n" +
+ "USE INFORMATION_SCHEMA\n" +
+ "DROP TABLE IF EXISTS `VERSION`\n" +
+ "%s\n" +
+ "SELECT 'Finished upgrading MetaStore schema to %s'\n";
+
+ @Override
+ protected Set<String> usedCommandLineArguments() {
+ return Sets.newHashSet("initSchemaTo", "dryRun");
+ }
+
+ @Override
+ protected void execute(TaskContext context) throws HiveMetaException {
+ String fromVersion = getFromVersion(context);
+
+ SchemaInfo schemaInfo = context.getSchemaInfo();
+ String toVersion = context.getCommandLine().getOptionValue("initSchemaTo");
+ if (StringUtils.isBlank(toVersion)) {
+ toVersion = SchemaInfo.getRequiredHiveSchemaVersion();
+ }
+
+ if (toVersion.equals(fromVersion)) {
+ System.out.println("No schema upgrade required from version " + fromVersion);
Review Comment:
Why it is println and not LOG?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1532799108
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1535844923
[Schematool with liquibase developers' guide.pdf](https://github.com/apache/hive/files/11404126/Schematool.with.liquibase.developers.guide.pdf)
[Schematool with Liquibase design document.pdf](https://github.com/apache/hive/files/11404128/Schematool.with.Liquibase.design.document.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rkirtir commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "rkirtir (via GitHub)" <gi...@apache.org>.
rkirtir commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1213918496
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveTaskProvider.java:
##########
Review Comment:
Can we consider creating enum for commnad type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1213948983
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveTaskProvider.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTaskProvider;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+public class HiveTaskProvider implements SchemaToolTaskProvider {
+
+ /**
+ * The map contains {@link Supplier} lambdas, so only the required {@link SchemaToolTask}s are instantiated.
+ */
+ private final Map<String, Supplier<SchemaToolTask>> taskSuppliers = new HashMap<>();
+
+ @Override
+ public SchemaToolTask getTask(String command) {
+ return taskSuppliers.getOrDefault(command, () -> null).get();
+ }
+
+ @Override
+ public Set<String> getSupportedDatabases() {
+ return new HashSet<>(Collections.singletonList(HiveSchemaHelper.DB_HIVE));
+ }
+
+ public HiveTaskProvider(SchemaToolTaskProvider embeddedHmsTaskProvider) {
+ taskSuppliers.put("initSchema", () -> new HiveContextTask().addChild(new HiveUpdateTask().addChild(embeddedHmsTaskProvider.getTask("info"))));
Review Comment:
I think strings like "initSchema", "initSchemaTo" etc should be constants. Since the same string is used elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1523269813
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [5 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [217 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1509477593
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [201 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1106964285
##########
beeline/src/test/org/apache/hive/beeline/schematool/TestHiveSchemaTool.java:
##########
@@ -49,7 +49,7 @@ public class TestHiveSchemaTool {
@Mock
private HiveConf hiveConf;
private HiveSchemaTool.HiveSchemaToolCommandBuilder builder;
- private String pasword = "reallySimplePassword";
+ private final String pasword = "reallySimplePassword";
Review Comment:
nit: Typo, variable name must be `password`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1509016469
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [203 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1533225549
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aturoczy commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221573881
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
Review Comment:
Hm interesting as I remember every stream was only responsible for there own lifecycle. In some patterns the streams are reusable. But if you say it, I believe you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221425890
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
Review Comment:
The file is created by the same user in the previous line, so I think it should have the necessary rights to delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27032: Introduce liquibase for HMS schema evolution [hive]
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1860948004
retest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1503379557
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [197 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1521901272
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [5 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [211 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1522776244
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [5 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [208 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221443995
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));
+ beeLine.getOpts().setSilent(true);
+ }
+ beeLine.getOpts().setAllowMultiLineCommand(false);
+ beeLine.getOpts().setIsolation(TRANSACTION_READ_COMMITTED);
+ // We can be pretty sure that an entire line can be processed as a single command since
+ // we always add a line separator at the end while calling dbCommandParser.buildCommand.
+ beeLine.getOpts().setEntireLineAsCommand(true);
+ LOG.debug("Going to run command <" + commandBuilder.buildToLog(sqlScriptFile) + ">");
+ int status = beeLine.begin(commandBuilder.buildToRun(sqlScriptFile), null, false);
+ if (status != 0) {
+ throw new IOException("Schema script failed, errorcode " + status);
+ }
+ }
+ }
+
+ public BeelineScriptExecutor(NestedScriptParser dbCommandParser, CommandBuilder commandBuilder) {
Review Comment:
Shame on me! :) This was the standard at my first two workplaces, and I got used to it.
https://imgflip.com/i/7oj55t
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1584693465
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [7 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [228 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1212044143
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -1228,7 +1228,7 @@ public static enum ConfVars {
+ "To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not "
+ "recommended for production use cases, run schematool command instead." ),
/**
- * @deprecated Use MetastoreConf.SCHEMA_VERIFICATION
+ * @deprecated No longer in user, neither {@link org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars#SCHEMA_VERIFICATION}
Review Comment:
nit: Typo in user, should be use
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/liquibase/XmlScriptScanner.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.liquibase;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This {@link ScriptScanner} implementation call can scan Liquibase XML format changesets for tables.
+ */
+class XmlScriptScanner implements ScriptScanner {
+
+ private final Pattern pattern = Pattern.compile("<createTable(?:.|\\s)*?tableName=\"(.+?)\"|<renameTable(?:.|\\s)*?oldTableName=\"(.+?)\"(?:.|\\s)*?newTableName=\"(.+?)\"|<dropTable(?:.|\\s)*?tableName=\"(.+?)\"");
+
+ /**
+ * @param dbType not in use
+ */
+ @Override
+ public void findTablesInScript(String scriptPath, String dbType, Set<String> tableList) throws HiveMetaException {
Review Comment:
dbType param is not used here, can this be removed
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(new NullOutputStream()));
+ beeLine.getOpts().setSilent(true);
+ }
+ beeLine.getOpts().setAllowMultiLineCommand(false);
+ beeLine.getOpts().setIsolation("TRANSACTION_READ_COMMITTED");
Review Comment:
We should create an enum/constant for this, WDYT @veghlaci05
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/SchemaInfo.java:
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+
+import java.io.File;
+import java.util.List;
+
+/**
+ * Provides information about the HMS or Hive schema
+ */
+public abstract class SchemaInfo {
+
+ /**
+ * This must be updated accordingly in case of breaking schema changes.
+ * For example: adding a new column to an HMS table.
+ */
+ private static final String MIN_HMS_SCHEMA_VERSION = "4.0.0";
Review Comment:
Should we make these configurable? @veghlaci05
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1495935984
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [12 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1494826493
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [12 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1213935645
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveUpdateTask.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import com.google.common.collect.Sets;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolCommandLine;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.TaskContext;
+import org.apache.hive.beeline.schematool.HiveSchemaTool;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+
+class HiveUpdateTask extends SchemaToolTask {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveUpdateTask.class);
+
+ private static final String VERSION_UPGRADE_SCRIPT =
+ "USE SYS\n" +
+ "DROP TABLE IF EXISTS `VERSION`\n" +
+ "%s\n" +
+ "USE INFORMATION_SCHEMA\n" +
+ "DROP TABLE IF EXISTS `VERSION`\n" +
+ "%s\n" +
+ "SELECT 'Finished upgrading MetaStore schema to %s'\n";
+
+ @Override
+ protected Set<String> usedCommandLineArguments() {
+ return Sets.newHashSet("initSchemaTo", "dryRun");
+ }
+
+ @Override
+ protected void execute(TaskContext context) throws HiveMetaException {
+ String fromVersion = getFromVersion(context);
+
+ SchemaInfo schemaInfo = context.getSchemaInfo();
+ String toVersion = context.getCommandLine().getOptionValue("initSchemaTo");
+ if (StringUtils.isBlank(toVersion)) {
+ toVersion = SchemaInfo.getRequiredHiveSchemaVersion();
+ }
+
+ if (toVersion.equals(fromVersion)) {
+ System.out.println("No schema upgrade required from version " + fromVersion);
+ return;
+ }
+
+ if (fromVersion.equals(HiveSchemaInfo.INITIAL_VERSION)) {
+ System.out.println("Initializing schema");
+ } else {
+ System.out.println("Starting upgrade metastore schema from version " + fromVersion + " to " + toVersion);
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ List<String> upgradeScripts = schemaInfo.getUnappliedScripts();
+
+ String scriptDir = schemaInfo.getMetaStoreScriptDir();
+ try {
+ for (String scriptFile : upgradeScripts) {
+ System.out.println("Upgrade script " + scriptFile);
+ if (!context.getCommandLine().hasOption("dryRun")) {
+ context.getScriptExecutor().execSql(scriptDir, scriptFile);
+ System.out.println("Completed " + scriptFile);
+ }
+ }
+
+ //Update schema version
+ File scriptFile = File.createTempFile("hiveVesionScript", "sql");
Review Comment:
Typo: hiveVersionScript
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveTaskProvider.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTaskProvider;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+public class HiveTaskProvider implements SchemaToolTaskProvider {
+
+ /**
+ * The map contains {@link Supplier} lambdas, so only the required {@link SchemaToolTask}s are instantiated.
+ */
+ private final Map<String, Supplier<SchemaToolTask>> taskSuppliers = new HashMap<>();
+
+ @Override
+ public SchemaToolTask getTask(String command) {
+ return taskSuppliers.getOrDefault(command, () -> null).get();
+ }
+
+ @Override
+ public Set<String> getSupportedDatabases() {
+ return new HashSet<>(Collections.singletonList(HiveSchemaHelper.DB_HIVE));
+ }
+
+ public HiveTaskProvider(SchemaToolTaskProvider embeddedHmsTaskProvider) {
+ taskSuppliers.put("initSchema", () -> new HiveContextTask().addChild(new HiveUpdateTask().addChild(embeddedHmsTaskProvider.getTask("info"))));
Review Comment:
I think strings like "initSchema", "initSchemaTo" etc should be constants should be constants. Since the same string is used elsewhere.
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveSchemaInfo.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+class HiveSchemaInfo extends SchemaInfo {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveSchemaInfo.class);
+ private static final String UPGRADE_FILE_PREFIX = "upgrade-";
+ protected static final String VERSION_UPGRADE_LIST = "upgrade.order";
+ protected static final String INITIAL_VERSION = "0.0.0";
+
+ String[] hiveSchemaVersions = null;
+
+ @Override
+ public List<String> getUnappliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ String schemaVersion = INITIAL_VERSION;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ } else throw e;
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+
+ // check if we are already at the required schema level
+ if (isVersionCompatible(getRequiredHiveSchemaVersion(), schemaVersion)) {
+ return upgradeScriptList;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ int firstScript = -1;
+ for (int i=0; i < hiveSchemaVersions.length; i++) {
+ if (hiveSchemaVersions[i].startsWith(schemaVersion + "-to-")) {
+ firstScript = i;
+ }
+ }
+ if (firstScript == -1) {
+ throw new HiveMetaException("Unknown version specified for upgrade " + schemaVersion + " Metastore schema may be too old or newer");
+ }
+
+ for (int i=firstScript; i < hiveSchemaVersions.length; i++) {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersions[i]);
+ upgradeScriptList.add(scriptFile);
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public List<String> getAppliedScripts() throws HiveMetaException {
+ if (hiveSchemaVersions == null) {
+ loadAllUpgradeScripts();
+ }
+ List <String> upgradeScriptList = new ArrayList<>();
+ String schemaVersion;
+ try {
+ schemaVersion = getSchemaVersion();
+ } catch (HiveMetaException e) {
+ if (e.getMessage().startsWith("Failed to get schema version")) {
+ LOG.warn("Unable to get schema version, assuming it's an empty schema.");
+ return upgradeScriptList;
+ } else throw e;
+ }
+
+ // Find the list of scripts to execute for this upgrade
+ for (String hiveSchemaVersion : hiveSchemaVersions) {
+ if (hiveSchemaVersion.startsWith(schemaVersion + "-to-")) {
+ break;
+ } else {
+ String scriptFile = generateUpgradeFileName(hiveSchemaVersion);
+ upgradeScriptList.add(scriptFile);
+ }
+ }
+ return upgradeScriptList;
+ }
+
+ @Override
+ public String getSchemaVersion() throws HiveMetaException {
+ try (Connection metastoreDbConnection = HiveSchemaHelper.getConnectionToMetastore(connectionInfo, conf, "SYS");
+ Statement stmt = metastoreDbConnection.createStatement()) {
+ ResultSet res = stmt.executeQuery("select t.SCHEMA_VERSION from VERSION t");
+ if (!res.next()) {
+ throw new HiveMetaException("Could not find version info in Hive VERSION table.");
+ }
+ String currentSchemaVersion = res.getString(1);
+ if (res.next()) {
+ throw new HiveMetaException("Multiple versions were found in version table.");
+ }
+ return currentSchemaVersion;
+ } catch (SQLException e) {
+ throw new HiveMetaException("Failed to get schema version, Cause:" + e.getMessage());
+ }
+ }
+
+ @Override
+ public String getCreateUserScript() throws HiveMetaException {
+ String createScript = CREATE_USER_PREFIX + "." + connectionInfo + SQL_FILE_EXTENSION;
+ File scriptFile = new File(getMetaStoreScriptDir() + File.separatorChar + createScript);
+ // check if the file exists
+ if (!scriptFile.exists()) {
+ throw new HiveMetaException("Unable to find create user file, expected: " + scriptFile.getAbsolutePath());
+ }
+ return createScript;
+ }
+
+ // format the upgrade script name eg upgrade-x-y-dbType.sql
+ private String generateUpgradeFileName(String fileVersion) {
+ return UPGRADE_FILE_PREFIX + fileVersion + "." + connectionInfo.getDbType() + SQL_FILE_EXTENSION;
+ }
+
+ private void loadAllUpgradeScripts() throws HiveMetaException {
+ // load upgrade order for the given dbType
+ List<String> upgradeOrderList = new ArrayList<>();
+ String upgradeListFile = getMetaStoreScriptDir() + File.separator + VERSION_UPGRADE_LIST + "." + connectionInfo.getDbType();
+ try (FileReader fr = new FileReader(upgradeListFile);
+ BufferedReader bfReader = new BufferedReader(fr)) {
+ String currSchemaVersion;
+ while ((currSchemaVersion = bfReader.readLine()) != null) {
+ upgradeOrderList.add(currSchemaVersion.trim());
+ }
+ } catch (FileNotFoundException e) {
+ throw new HiveMetaException("File " + upgradeListFile + " not found ", e);
+ } catch (IOException e) {
+ throw new HiveMetaException("Error reading " + upgradeListFile, e);
+ }
+ hiveSchemaVersions = upgradeOrderList.toArray(new String[0]);
+ }
+
+
+ public HiveSchemaInfo(String metastoreHome, HiveSchemaHelper.MetaStoreConnectionInfo connectionInfo, Configuration conf) throws HiveMetaException {
Review Comment:
Is HiveMetaException thrown here? I think this can be removed.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/commandparser/AbstractCommandParser.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.commandparser;
+
+import com.google.common.collect.Lists;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.IllegalFormatException;
+import java.util.List;
+
+/**
+ * Base implementation of NestedScriptParser abstractCommandParser.
+ */
+abstract class AbstractCommandParser implements NestedScriptParser {
+
+ private List<String> dbOpts;
+ // Depending on whether we are using beeline or sqlline the line endings have to be handled
+ // differently.
+ private final boolean usingSqlLine;
+
+ public AbstractCommandParser(String dbOpts, boolean usingSqlLine) {
+ setDbOpts(dbOpts);
+ this.usingSqlLine = usingSqlLine;
+ }
+
+ @Override
+ public boolean isPartialCommand(String dbCommand) throws IllegalArgumentException{
+ if (dbCommand == null || dbCommand.isEmpty()) {
+ throw new IllegalArgumentException("invalid command line " + dbCommand);
+ }
+ dbCommand = dbCommand.trim();
+ if (dbCommand.endsWith(getDelimiter()) || isNonExecCommand(dbCommand)) {
+ return false;
+ } else {
+ return true;
+ }
Review Comment:
This can be simplified to -
return !(dbCommand.endsWith(getDelimiter()) || isNonExecCommand(dbCommand));
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/liquibase/LiquibaseUpdateToTask.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.liquibase;
+
+import com.google.common.collect.Sets;
+import liquibase.Contexts;
+import liquibase.LabelExpression;
+import liquibase.Labels;
+import liquibase.Liquibase;
+import liquibase.changelog.ChangeSetStatus;
+import liquibase.exception.LiquibaseException;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.TaskContext;
+
+import java.io.PrintWriter;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * This task utilizes {@link Liquibase} to init or upgrade the HMS schema up to a specific version. Since Liquibase
+ * doesn't maintain a database version, it is not possible to do this migration out of the box. In order to be able to
+ * run only a part of the migration scripts, labels are used. Each script has a version label, and the labels having
+ * less or equal version than 'initSchemaTo' are passed to {@link Liquibase#update(Contexts, LabelExpression)} as inlcusive
+ * filters. This requires that labels must be a parseable valid version
+ * (verified by {@link org.apache.hadoop.hive.metastore.SchemaInfo#isValidVersion(String)}).
+ */
+class LiquibaseUpdateToTask extends SchemaToolTask {
+
+ @Override
+ protected Set<String> usedCommandLineArguments() {
+ return Sets.newHashSet("initSchemaTo", "dryRun");
+ }
+
+ @Override
+ protected void execute(TaskContext context) throws HiveMetaException {
+ try {
+ String toVersion = context.getCommandLine().getOptionValue("initSchemaTo");
+
+ Liquibase liquibase = context.getLiquibase();
+ Contexts liquibaseContexts = context.getLiquibaseContext();
+
+ liquibase.getLog().info("Starting metastore schema upgrade to version: " + toVersion);
+
+ List<ChangeSetStatus> unappliedChanges = liquibase
+ .getChangeSetStatuses(liquibaseContexts, null, false)
+ .stream()
+ .filter(s -> !s.getPreviouslyRan())
+ .collect(Collectors.toList());
+
+ StringBuilder logEntry = new StringBuilder("The following scripts will be applied: ");
+ StringBuilder labelFilter = new StringBuilder();
+ boolean foundToVersion = false;
+ for (int i = 0; i < unappliedChanges.size(); i++) {
+ logEntry.append(unappliedChanges.get(i).getChangeSet().getFilePath());
+
+ Labels currentVerison = unappliedChanges.get(i).getChangeSet().getLabels();
+
+ if (labelFilter.length() > 0) {
+ labelFilter.append(" OR ");
+ logEntry.append(", ");
+ }
+
+ labelFilter.append(String.join(" OR ", currentVerison.getLabels()));
+
+ if (currentVerison.getLabels().contains(toVersion)) {
+ foundToVersion = true;
+ break;
+ }
+ }
+ if (!foundToVersion) {
+ throw new HiveMetaException("The required version (" + toVersion + ") could not be found among the version scripts!");
+ }
+ liquibase.getLog().info(logEntry.toString());
+
+ if (context.getCommandLine().hasOption("dryRun")) {
+ liquibase.update(liquibaseContexts, new LabelExpression(labelFilter.toString()), new PrintWriter(System.out));
+ } else {
+ liquibase.update(liquibaseContexts, new LabelExpression(labelFilter.toString()));
+ }
+
+ liquibase.getLog().info("Metastore schema upgraded to version: " + toVersion);
+ } catch (LiquibaseException e) {
+ throw new HiveMetaException("Schema upgrade FAILED! Metastore state would be inconsistent!", e);
Review Comment:
I see this string at multiple places. Ideal to add this as a constant.
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(new NullOutputStream()));
Review Comment:
Creating NullOutputStream is deprecated. Can we use the singleton offered by NullOutputStream.
`beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1580421483
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [209 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aturoczy commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221570544
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));
+ beeLine.getOpts().setSilent(true);
+ }
+ beeLine.getOpts().setAllowMultiLineCommand(false);
+ beeLine.getOpts().setIsolation(TRANSACTION_READ_COMMITTED);
+ // We can be pretty sure that an entire line can be processed as a single command since
+ // we always add a line separator at the end while calling dbCommandParser.buildCommand.
+ beeLine.getOpts().setEntireLineAsCommand(true);
+ LOG.debug("Going to run command <" + commandBuilder.buildToLog(sqlScriptFile) + ">");
+ int status = beeLine.begin(commandBuilder.buildToRun(sqlScriptFile), null, false);
+ if (status != 0) {
+ throw new IOException("Schema script failed, errorcode " + status);
+ }
+ }
+ }
+
+ public BeelineScriptExecutor(NestedScriptParser dbCommandParser, CommandBuilder commandBuilder) {
Review Comment:
Deserved it! :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] aturoczy commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "aturoczy (via GitHub)" <gi...@apache.org>.
aturoczy commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221568567
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
Review Comment:
ACK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1431166190
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [10 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [205 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1494447775
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [12 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [207 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1511361599
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [202 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1513354516
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [205 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: [Draft] Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1517664678
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [211 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4060:
URL: https://github.com/apache/hive/pull/4060#issuecomment-1535946112
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4060)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT) [6 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4060&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL) [210 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4060&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4060&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rkirtir commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "rkirtir (via GitHub)" <gi...@apache.org>.
rkirtir commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1213919960
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/HiveUpdateTask.java:
##########
Review Comment:
Are we considering seperate exception type/code related to schema change/upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1213944337
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
+ try (BufferedWriter out = new BufferedWriter(fstream)) {
+ if (!commandBuilder.getConnectionInfo().getDbType().equalsIgnoreCase(HiveSchemaHelper.DB_HIVE)) {
+ out.write("!autocommit off" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ out.write("!commit" + System.getProperty(LINE_SEPARATOR));
+ } else {
+ out.write("!autocommit on" + System.getProperty(LINE_SEPARATOR));
+ out.write(sqlCommands);
+ }
+ out.write("!closeall" + System.getProperty(LINE_SEPARATOR));
+ }
+ execSql(tmpFile.getPath());
+ }
+
+ public void execSql(String sqlScriptFile) throws IOException {
+ // run the script using Beeline
+ try (BeeLine beeLine = new BeeLine()) {
+ if (!commandBuilder.isVerbose()) {
+ beeLine.setOutputStream(new PrintStream(new NullOutputStream()));
Review Comment:
Creating NullOutputStream object is deprecated. Can we use the singleton offered by NullOutputStream.
`beeLine.setOutputStream(new PrintStream(NullOutputStream.NULL_OUTPUT_STREAM));`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rkirtir commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "rkirtir (via GitHub)" <gi...@apache.org>.
rkirtir commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1214018916
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/liquibase/LiquibaseValidationTask.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.liquibase;
+
+import liquibase.Contexts;
+import liquibase.Labels;
+import liquibase.Liquibase;
+import liquibase.changelog.ChangeSetStatus;
+import liquibase.exception.LiquibaseException;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.SchemaInfo;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.SchemaToolTask;
+import org.apache.hadoop.hive.metastore.tools.schematool.task.TaskContext;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+class LiquibaseValidationTask extends SchemaToolTask {
+
+ @Override
+ protected Set<String> usedCommandLineArguments() {
+ return null;
+ }
+
+ @Override
+ @SuppressWarnings({ "squid:S2201", "ResultOfMethodCallIgnored" })
+ protected void execute(TaskContext context) throws HiveMetaException {
+ Liquibase liquibase = context.getLiquibase();
+ Contexts liquibaseContexts = context.getLiquibaseContext();
+
+ liquibase.getLog().info("Validating liquibase changesets if match all requirements.");
+
+ try {
+ List<ChangeSetStatus> statuses = liquibase.getChangeSetStatuses(liquibaseContexts, null, false);
+
+ for (int i = 1; i < statuses.size(); i++) {
+ ChangeSetStatus status = statuses.get(i);
+ Labels version = status.getChangeSet().getLabels();
+ if (version == null || version.isEmpty()) {
+ throw new HiveMetaException("All version scripts must have a label containing one or more valid version(s)! " +
+ "The following script does not have any label: " + status.getChangeSet().getFilePath());
+ }
+
+ if (version.getLabels().stream().anyMatch(s -> !SchemaInfo.isValidVersion(version.toString()))) {
+ throw new HiveMetaException("All version scripts must have a label containing one or more valid version(s)! " +
+ "Valid versions are consist of major, minor, incremental version numbers and optionally can have qualifiers. " +
+ "The label ( " + version + ") of the following script does not conform the required format: " +
+ status.getChangeSet().getFilePath());
+ }
+
+ if (!isInteger(status.getChangeSet().getId())) {
+ throw new HiveMetaException("All changesets must have a vaild and unique integer id! The following changeset " +
+ "does not have a valid integer id: " + status.getChangeSet().getFilePath());
+ }
+ if (StringUtils.isBlank(status.getChangeSet().getComments())) {
+ throw new HiveMetaException("All changesets must have a vaild comment! The following changeset " +
+ "does not have a valid comment: " + status.getChangeSet().getFilePath());
+ }
+ }
+ //No need to check the result, Collectors.toMap will fail with IllegalStateException in case of duplicate key(s).
+ String dbType = HiveSchemaHelper.DB_POSTGRACE.equals(context.getCommandLine().getDbType())
+ ? "postgresql"
+ : context.getCommandLine().getDbType();
+ statuses.stream()
+ .filter(s -> CollectionUtils.isEmpty(s.getChangeSet().getDbmsSet()) || s.getChangeSet().getDbmsSet().contains(dbType))
+ .collect(Collectors.toMap(s -> s.getChangeSet().getId(), s -> s));
+ } catch (LiquibaseException e) {
+ throw new HiveMetaException("Unable obtain the liquibase changelog.", e);
+ } catch (IllegalStateException e) {
+ if (e.getMessage().startsWith("Duplicate key")) {
+ throw new HiveMetaException("All changesets must have a unique id. One or more changesets have the same id!");
+ }
+ throw new HiveMetaException("Unable obtain the liquibase changelog.", e);
+ }
+ liquibase.getLog().info("Validation of Liquibase changesets successfull.");
+ }
+
+ private boolean isInteger(String strNum) {
+ if (StringUtils.isBlank(strNum)) {
Review Comment:
Can we consider NumberUtils.isParsable(strNum) instead of this block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221431338
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
+ tmpFile.deleteOnExit();
+
+ // write out the buffer into a file. Add beeline commands for autocommit and close
+ FileWriter fstream = new FileWriter(tmpFile.getPath());
Review Comment:
The writes is wrapped by a `BufferedWriter`, which is in a _try-with-resources_ block. The `BufferedWriter` delegates the close call to the underlying `FileWriter`, so there is no need to close it explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1221424290
##########
beeline/src/java/org/apache/hive/beeline/schematool/tasks/BeelineScriptExecutor.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.schematool.tasks;
+
+import org.apache.commons.io.output.NullOutputStream;
+import org.apache.hadoop.hive.metastore.tools.schematool.CommandBuilder;
+import org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParser;
+import org.apache.hadoop.hive.metastore.tools.schematool.scriptexecution.ScriptExecutor;
+import org.apache.hive.beeline.BeeLine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintStream;
+
+/**
+ * {@link BeeLine} based {@link ScriptExecutor} implementation. Able to execute the given scripts using {@link BeeLine} internally.
+ * Can be used to execute scripts against the Hive schema.
+ */
+class BeelineScriptExecutor implements ScriptExecutor {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeelineScriptExecutor.class);
+ private static final String LINE_SEPARATOR = "line.separator";
+
+ private final NestedScriptParser dbCommandParser;
+ private final CommandBuilder commandBuilder;
+
+ @Override
+ public void execSql(String scriptDir, String sqlScriptFile) throws IOException {
+ // expand the nested script
+ // If the metaDbType is set, this is setting up the information
+ // schema in Hive. That specifically means that the sql commands need
+ // to be adjusted for the underlying RDBMS (correct quotation
+ // strings, etc).
+ String sqlCommands = dbCommandParser.buildCommand(scriptDir, sqlScriptFile, true);
+ File tmpFile = File.createTempFile("schematool", ".sql");
Review Comment:
This file is temporary, it is created using the original script file, and modified in a way so Beeline can execute it properly. Once the script is executed it is no longer needed. `File.deleteOnExit()` ensures that the file will be deleted upon JVM exit (Since SchemaTool is a CLI tool we are talking about seconds, or minutes at max).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1218014820
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/task/SchemaToolTaskFactory.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.task;
+
+import org.apache.commons.cli.Option;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolCommandLine;
+import org.apache.hadoop.hive.metastore.tools.schematool.commandparser.NestedScriptParserFactory;
+
+import java.util.Set;
+
+/**
+ * Responsible for creating the {@link SchemaToolTask} chain based on the input provided via the {@link SchemaToolCommandLine}
+ * object
+ */
+public class SchemaToolTaskFactory {
+
+ private final SchemaToolTaskProvider[] providers;
+ private final NestedScriptParserFactory scriptParserFactory;
+
+ /**
+ * Creates the {@link SchemaToolTask} chain based on the input provided via the {@link SchemaToolCommandLine}.
+ * Checks for dangling command line arguments (arguments which are not used by any of the tasks in the created task chain).
+ * @param cmdLine A {@link SchemaToolCommandLine} instance containing the parsed command line arguments.
+ * @return Returns with the created {@link SchemaToolTask} chain.
+ * @throws HiveMetaException Thrown in the following cases:
+ * <ul>
+ * <li>Dangling arguments found</li>
+ * <li>No task found for the given command line arguments</li>
+ * <li>None of the available {@link SchemaToolTaskProvider} instances are supporting the database type provided via the
+ * '-dbType' option. For list of supported DB types see: {@link org.apache.hadoop.hive.metastore.tools.schematool.HiveSchemaHelper} constants</li>
+ * </ul>
+ */
+ public SchemaToolTask getTask(SchemaToolCommandLine cmdLine) throws HiveMetaException {
+ for(SchemaToolTaskProvider provider : providers) {
+ if(!provider.getSupportedDatabases().contains(cmdLine.getDbType().toLowerCase())) {
+ continue;
+ }
+ SchemaToolTask rootTask = new RootTask(scriptParserFactory);
+ for (Option command : cmdLine.getOptions()) {
+ SchemaToolTask task = provider.getTask(command.getOpt());
+ if (task != null) {
+ rootTask.addChild(task);
+ //Check for unused command line arguments
+ Set<String> arguments = rootTask.getUsedCommandLineArguments();
+ arguments.add(command.getOpt());
+ arguments = cmdLine.getDanglingArguments(arguments);
+ if (arguments.size() > 0) {
+ throw new HiveMetaException("The following arguments are not used by the given command (" + command + "): " + String.join(",", arguments));
Review Comment:
Having an unused argument should not cause any issue, however this was the original behavior which I did not want to change as there may scripts depend on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] veghlaci05 commented on a diff in pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4060:
URL: https://github.com/apache/hive/pull/4060#discussion_r1217989288
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/liquibase/XmlScriptScanner.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.tools.schematool.liquibase;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.HiveMetaException;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This {@link ScriptScanner} implementation call can scan Liquibase XML format changesets for tables.
+ */
+class XmlScriptScanner implements ScriptScanner {
+
+ private final Pattern pattern = Pattern.compile("<createTable(?:.|\\s)*?tableName=\"(.+?)\"|<renameTable(?:.|\\s)*?oldTableName=\"(.+?)\"(?:.|\\s)*?newTableName=\"(.+?)\"|<dropTable(?:.|\\s)*?tableName=\"(.+?)\"");
+
+ /**
+ * @param dbType not in use
+ */
+ @Override
+ public void findTablesInScript(String scriptPath, String dbType, Set<String> tableList) throws HiveMetaException {
Review Comment:
Unfortunately no, the signature is coming from the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27032: Introduce liquibase for HMS schema evolution [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4060: HIVE-27032: Introduce liquibase for HMS schema evolution
URL: https://github.com/apache/hive/pull/4060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org