You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/02/23 17:59:53 UTC
[GitHub] [accumulo] keith-turner commented on a change in pull request #1946: re #1451: Initial external compaction thrift APIs
keith-turner commented on a change in pull request #1946:
URL: https://github.com/apache/accumulo/pull/1946#discussion_r581257653
##########
File path: core/src/main/thrift/compaction-coordinator.thrift
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+namespace java org.apache.accumulo.core.compaction.thrift
+namespace cpp org.apache.accumulo.core.compaction.thrift
+
+include "data.thrift"
+include "security.thrift"
+include "tabletserver.thrift"
+include "trace.thrift"
+
+enum Action {
+ ADD
+ UPDATE
+ CANCEL
+}
+
+enum CompactionState {
+ # Coordinator should set state to ASSIGNED when getCompactionJob is called by Compactor
+ ASSIGNED
+ # Compactor should set state to STARTED when compaction has successfully begun
+ STARTED
+ # Compactor can call repeatedly with an updated message to reflect percentage complete
+ IN_PROGRESS
+ # Compactor should set state to SUCCEEDED when compaction job has successfully finished
+ SUCCEEDED
+ # Compactor should set state to FAILED when compaction job fails, message should be mandatory
+ FAILED
+ # Compactor should set state to CANCELLED to acknowledge that it has stopped compacting
+ CANCELLED
+}
+
+struct CompactionJob {
+ 1:trace.TInfo traceInfo
+ 2:security.TCredentials credentials
+ 3:i64 compactionId
+ 5:data.TKeyExtent extent
+ 6:list<string> files
+ 7:i32 priority
+ 8:i32 readRate
+ 9:i32 writeRate
+ 10:tabletserver.IteratorConfig iteratorSettings
+ 11:tabletserver.CompactionType type
+ # Need to add SELECTOR To CompactionReason, delete CompactionKind?
+ 12:tabletserver.CompactionReason reason
+}
+
+struct Status {
+ 1:i64 timestamp
+ 2:i64 compactionId
+ 3:string compactorHostAndPort
+ 4:CompactionState state
+ 5:string message
+}
+
+service CompactionCoordinator {
+
+ /*
+ * Called by TabletServer (or CLI) to add, update, or cancel a compaction for a tablet
+ */
+ void AddOrUpdateCompaction(
+ 1:string tableId
+ 2:string endRow
Review comment:
Will probably end up needed to use TKeyExtent so that we can distinguish tablets that have split or merged.
##########
File path: core/src/main/thrift/compaction-coordinator.thrift
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+namespace java org.apache.accumulo.core.compaction.thrift
+namespace cpp org.apache.accumulo.core.compaction.thrift
+
+include "data.thrift"
+include "security.thrift"
+include "tabletserver.thrift"
+include "trace.thrift"
+
+enum Action {
+ ADD
+ UPDATE
+ CANCEL
+}
+
+enum CompactionState {
+ # Coordinator should set state to ASSIGNED when getCompactionJob is called by Compactor
+ ASSIGNED
+ # Compactor should set state to STARTED when compaction has successfully begun
+ STARTED
+ # Compactor can call repeatedly with an updated message to reflect percentage complete
+ IN_PROGRESS
+ # Compactor should set state to SUCCEEDED when compaction job has successfully finished
+ SUCCEEDED
+ # Compactor should set state to FAILED when compaction job fails, message should be mandatory
+ FAILED
+ # Compactor should set state to CANCELLED to acknowledge that it has stopped compacting
+ CANCELLED
+}
+
+struct CompactionJob {
+ 1:trace.TInfo traceInfo
+ 2:security.TCredentials credentials
+ 3:i64 compactionId
+ 5:data.TKeyExtent extent
+ 6:list<string> files
+ 7:i32 priority
+ 8:i32 readRate
+ 9:i32 writeRate
+ 10:tabletserver.IteratorConfig iteratorSettings
+ 11:tabletserver.CompactionType type
+ # Need to add SELECTOR To CompactionReason, delete CompactionKind?
+ 12:tabletserver.CompactionReason reason
+}
+
+struct Status {
+ 1:i64 timestamp
+ 2:i64 compactionId
+ 3:string compactorHostAndPort
+ 4:CompactionState state
+ 5:string message
+}
+
+service CompactionCoordinator {
+
+ /*
+ * Called by TabletServer (or CLI) to add, update, or cancel a compaction for a tablet
+ */
+ void AddOrUpdateCompaction(
+ 1:string tableId
+ 2:string endRow
+ 3:string queueName
+ 4:i64 priority
+ 5:Action action
+ )
+
+ /*
+ * Called by TabletServer (or CLI) to get current status of compaction for a tablet
+ */
+ Status getCompactionStatus(
Review comment:
How do you envision this being used?
##########
File path: core/src/main/thrift/compaction-coordinator.thrift
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+namespace java org.apache.accumulo.core.compaction.thrift
+namespace cpp org.apache.accumulo.core.compaction.thrift
+
+include "data.thrift"
+include "security.thrift"
+include "tabletserver.thrift"
+include "trace.thrift"
+
+enum Action {
+ ADD
+ UPDATE
+ CANCEL
+}
+
+enum CompactionState {
+ # Coordinator should set state to ASSIGNED when getCompactionJob is called by Compactor
+ ASSIGNED
+ # Compactor should set state to STARTED when compaction has successfully begun
+ STARTED
+ # Compactor can call repeatedly with an updated message to reflect percentage complete
+ IN_PROGRESS
+ # Compactor should set state to SUCCEEDED when compaction job has successfully finished
+ SUCCEEDED
+ # Compactor should set state to FAILED when compaction job fails, message should be mandatory
+ FAILED
+ # Compactor should set state to CANCELLED to acknowledge that it has stopped compacting
+ CANCELLED
+}
+
+struct CompactionJob {
+ 1:trace.TInfo traceInfo
+ 2:security.TCredentials credentials
+ 3:i64 compactionId
+ 5:data.TKeyExtent extent
+ 6:list<string> files
+ 7:i32 priority
+ 8:i32 readRate
+ 9:i32 writeRate
+ 10:tabletserver.IteratorConfig iteratorSettings
+ 11:tabletserver.CompactionType type
+ # Need to add SELECTOR To CompactionReason, delete CompactionKind?
+ 12:tabletserver.CompactionReason reason
+}
+
+struct Status {
+ 1:i64 timestamp
+ 2:i64 compactionId
+ 3:string compactorHostAndPort
+ 4:CompactionState state
+ 5:string message
+}
+
+service CompactionCoordinator {
+
+ /*
+ * Called by TabletServer (or CLI) to add, update, or cancel a compaction for a tablet
+ */
+ void AddOrUpdateCompaction(
+ 1:string tableId
+ 2:string endRow
+ 3:string queueName
+ 4:i64 priority
+ 5:Action action
+ )
+
+ /*
+ * Called by TabletServer (or CLI) to get current status of compaction for a tablet
+ */
+ Status getCompactionStatus(
+ 1:string tableId
+ 2:string endRow
+ 3:string queueName
+ )
+
+ /*
+ * Called by TabletServer (or CLI) to get complete status of compaction for a tablet
+ */
+ list<Status> getCompactionStatusHistory(
Review comment:
How do you envision this being used?
##########
File path: core/src/main/thrift/compaction-coordinator.thrift
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+namespace java org.apache.accumulo.core.compaction.thrift
+namespace cpp org.apache.accumulo.core.compaction.thrift
+
+include "data.thrift"
+include "security.thrift"
+include "tabletserver.thrift"
+include "trace.thrift"
+
+enum Action {
+ ADD
+ UPDATE
+ CANCEL
+}
+
+enum CompactionState {
+ # Coordinator should set state to ASSIGNED when getCompactionJob is called by Compactor
+ ASSIGNED
+ # Compactor should set state to STARTED when compaction has successfully begun
+ STARTED
+ # Compactor can call repeatedly with an updated message to reflect percentage complete
+ IN_PROGRESS
+ # Compactor should set state to SUCCEEDED when compaction job has successfully finished
+ SUCCEEDED
+ # Compactor should set state to FAILED when compaction job fails, message should be mandatory
+ FAILED
+ # Compactor should set state to CANCELLED to acknowledge that it has stopped compacting
+ CANCELLED
+}
+
+struct CompactionJob {
+ 1:trace.TInfo traceInfo
+ 2:security.TCredentials credentials
+ 3:i64 compactionId
+ 5:data.TKeyExtent extent
+ 6:list<string> files
Review comment:
This will also need an output file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org