You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/07/19 17:28:51 UTC
[GitHub] [hive] adesh-rao opened a new pull request, #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
adesh-rao opened a new pull request, #3456:
URL: https://github.com/apache/hive/pull/3456
HiveSplitGenerator is tightly coupled with Tez's InputContext to fetch the available slots during split calculation. Creating a interface to fetch the available slots will allow having other implementation too which are more suitable for different cloud vendors.
The change contains a default implementation using Tez's InputContext and a new HiveConfiguration pointing to the class implementing the interface. Different cloud vendors can plugin there strategy by implementing the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3456:
URL: https://github.com/apache/hive/pull/3456#discussion_r925239340
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezAvailableSlotsCalculator.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.runtime.api.InputInitializerContext;
+
+/**
+ * Default implementation of AvailableSlotsCalculator which relies on available capacity of the cluster
+ */
+public class TezAvailableSlotsCalculator implements AvailableSlotsCalculator {
+ @Override
+ public void initialize(Configuration conf) {
+ // Noop
+ }
+
+ @Override
+ public int getAvailaleSlots(InputInitializerContext inputInitializerContext) {
Review Comment:
typo: availaleSlots
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AvailableSlotsCalculator.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.runtime.api.InputInitializerContext;
+
+/**
+ * Interface for fetching available slots during split calculation
+ */
+public interface AvailableSlotsCalculator {
+
+ void initialize(Configuration conf);
+
+ int getAvailaleSlots(InputInitializerContext inputInitializerContext);
Review Comment:
@adesh-rao : could you please describe this scenario and a possible implementation class further?
jira description implies that you want to get rid of tightly coupled tez input context, but the interface has tez InputInitializerContext in the signature, so I don't think we addressed the original description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on pull request #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on PR #3456:
URL: https://github.com/apache/hive/pull/3456#issuecomment-1194924210
@abstractdog Please can you review this request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh merged pull request #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
Posted by GitBox <gi...@apache.org>.
sankarh merged PR #3456:
URL: https://github.com/apache/hive/pull/3456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a diff in pull request #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on code in PR #3456:
URL: https://github.com/apache/hive/pull/3456#discussion_r925332287
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezAvailableSlotsCalculator.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.runtime.api.InputInitializerContext;
+
+/**
+ * Default implementation of AvailableSlotsCalculator which relies on available capacity of the cluster
+ */
+public class TezAvailableSlotsCalculator implements AvailableSlotsCalculator {
+ @Override
+ public void initialize(Configuration conf) {
+ // Noop
+ }
+
+ @Override
+ public int getAvailaleSlots(InputInitializerContext inputInitializerContext) {
Review Comment:
Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a diff in pull request #3456: HIVE-26412: Create interface to fetch available slots and add the default implementation
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on code in PR #3456:
URL: https://github.com/apache/hive/pull/3456#discussion_r925333209
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AvailableSlotsCalculator.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.tez.runtime.api.InputInitializerContext;
+
+/**
+ * Interface for fetching available slots during split calculation
+ */
+public interface AvailableSlotsCalculator {
+
+ void initialize(Configuration conf);
+
+ int getAvailaleSlots(InputInitializerContext inputInitializerContext);
Review Comment:
Removed the dependency of tez input context on the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org