[GitHub] [accumulo] dlmarion commented on issue #3658: Convert Compactor/Coordinator into generic distributed job execution service

["dlmarion (via GitHub)" <gi...@apache.org>]

dlmarion commented on issue #3658:
URL: https://github.com/apache/accumulo/issues/3658#issuecomment-1666177110

   I have been doing research trying to find a cross-language solution so that others can write task workers in other languages or frameworks. The problem that I'm finding is that I can't create a generic message type, but then enforce specific attribute values in certain cases.
   
   A Thrift example of this would be something like:
   ```
   enum TaskMessageType {
     COMPACTION_JOB_REQUEST,
     COMPACTION_JOB_RESPONSE,
     COMPACTION_STATUS_MSG,
     COMPACTION_COMPLETED_MSG,
     COMPACTION_FAILED_MSG
     COMPACTIONS_COMPLETED_RESPONSE
   }
   
   struct TaskMessage {
     1:TaskMessageType messageType
     1:binary serializedMessage
   }
   
   service TaskManager {
   
     TaskObject getTask(
       1:client.TInfo tinfo
       2:security.TCredentials credentials  
       3:TaskMessage message
       4:string taskID  
     )
   
     void taskCompleted(
       1:client.TInfo tinfo
       2:security.TCredentials credentials  
       3:TaskMessage message
     )
   
     void taskFailed(
       1:client.TInfo tinfo
       2:security.TCredentials credentials  
       3:TaskMessage message
     )
     
   }
   ```
   With this structure above, there is no way to enforce that when `taskCompleted` is called that the TaskMessageType is `COMPACTION_COMPLETED_MSG` in the TaskMessage. Even if we were to move the enum out of TaskMessage and add it as a field on `taskFailed`, there is no way to enforce that it has that value. In addition to Thrift, I looked at GRPC which uses ProtocolBuffers, and I don't see a way to do it there either. Obviously if we write the server component we can enforce the message type in the service method implementation. But if someone were to write a worker implementation in Go for example, it would not be clear to them from the Thrift IDL what message types are acceptable for the different service methods. My initial thought is to abandon the idea of supporting cross-language or user-supplied workers in any easy manner. I think that I can use a structure like this for the manager and worker implementations that we write. Thoughts on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org