You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "JeffBolle (via GitHub)" <gi...@apache.org> on 2023/07/27 02:39:58 UTC

[GitHub] [pulsar] JeffBolle opened a new issue, #20890: Add ElasticSearch BatchSource

JeffBolle opened a new issue, #20890:
URL: https://github.com/apache/pulsar/issues/20890

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Motivation
   
   ElasticSearch and Pulsar are very different types of data storage systems, however, there may be occasions where large amount of data that had been previously stored in ElasticSearch needs to be read into a Pulsar topic. We are working on transitioning a large data storage and search system which has tens of billions of records and hundreds of TB of data that would better serve the organization in Pulsar.  There are likely other use cases where data is being ingested to ElasticSearch but also needs to be loaded into Pulsar at scale.  The goal of this is not change detection, but simply to run a query and stream the results to a topic in Pulsar in a highly parallel way that will allow for copying very large datasets to Pulsar topics.  
   
   ### Solution
   
   Implement a BatchSource that is able to leverage the existing ElasticSearchSink code to build an ElasticSearch client, conduct a search that can be parallelized and send the results to a Pulsar topic.  The motivation behind using the BatchSource is to use the `discover` phase to create and distribute the search slices, allowing the user to create a sliced search to be executed in parallel with multiple function runners. The user will be able to choose to run a Scroll or PIT search depending on what their ElasticSearch version supports.
   
   ### Alternatives
   
   _No response_
   
   ### Anything else?
   
   I'm working on the PR implementing this feature.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add ElasticSearch BatchSource [pulsar]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20890:
URL: https://github.com/apache/pulsar/issues/20890#issuecomment-1751889677

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] JeffBolle commented on issue #20890: Add ElasticSearch BatchSource

Posted by "JeffBolle (via GitHub)" <gi...@apache.org>.
JeffBolle commented on issue #20890:
URL: https://github.com/apache/pulsar/issues/20890#issuecomment-1709418414

   I'm working on it. I had some vacations and other work items come up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add ElasticSearch BatchSource [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #20890: Add ElasticSearch BatchSource
URL: https://github.com/apache/pulsar/issues/20890


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #20890: Add ElasticSearch BatchSource

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20890:
URL: https://github.com/apache/pulsar/issues/20890#issuecomment-1694543897

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add ElasticSearch BatchSource [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #20890:
URL: https://github.com/apache/pulsar/issues/20890#issuecomment-6388791662

   Thanks @JeffBolle. The built-in connectors, including the Elasticsearch sink this would build on, have moved to the separate apache/pulsar-connectors repository (PIP-465), so this change can no longer be made here. If you're still interested, please open the request or a PR in https://github.com/apache/pulsar-connectors. Closing here as not planned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org