[jira] [Comment Edited] (TIKA-4181) Grpc + Tika Pipes - pipe iterator and emitter

["Nicholas DiPiazza (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805762#comment-17805762 ] 

Nicholas DiPiazza edited comment on TIKA-4181 at 1/11/24 6:25 PM:
------------------------------------------------------------------

Tika pipes could get a full fledged service that could be tika-server-http2 to accompany tika-server and maybe one day replace it? 

Not sure the best way to handle packaging the app, but we could create a secondary main method for running the tika-pipes as a grpc service.

Then we would create a protobuf contract for each of the new services that we do:
 * pipe crud operations - create, update, delete, read, list, etc
 * run a pipe job - takes bidirectional streams of data - incoming=fetch metadata objects, outgoing=emitDocuments
 ** this will use a configured fetcher

So you would then provide a Go example and Java example generated from our protobuf schema.  that people could take and use

 

 


was (Author: ndipiazza):
Tika pipes could get a full fledged service that could be tika-server-http2 to accompany tika-server and maybe one day replace it? 

Not sure the best way to handle packaging the app, but we could create a secondary main method for running the tika-pipes as a grpc service.

Then we would create a protobuf contract for each of the new services that we do:
 * pipe crud operations - create, update, delete, read, list, etc
 * run a pipe job - takes bidirectional streams of data - incoming=fetch metadata objects, outgoing=emitDocuments

So you would then provide a Go example and Java example generated from our protobuf schema.  that people could take and use

 

 

> Grpc + Tika Pipes - pipe iterator and emitter
> ---------------------------------------------
>
>                 Key: TIKA-4181
>                 URL: https://issues.apache.org/jira/browse/TIKA-4181
>             Project: Tika
>          Issue Type: New Feature
>          Components: tika-pipes
>            Reporter: Nicholas DiPiazza
>            Priority: Major
>
> Add full tika-pipes support of grpc
>  * pipe iterator
>  * fetcher
>  * emitter
> Requires we create a service contract that specifies the inputs we require from each method.
> Then we will need to implement the different components with a grpc client generated using the contract.
> This would enable developers to run tika-pipes as a persistently running daemon instead of just a single batch app, because it can continue to stream out more inputs.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)