You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "epugh (via GitHub)" <gi...@apache.org> on 2025/01/06 13:14:33 UTC

[PR] Demonstrate roundtrip export/import works [solr]

epugh opened a new pull request, #2940:
URL: https://github.com/apache/solr/pull/2940

   https://issues.apache.org/jira/browse/SOLR-13689
   
   
   # Description
   
   Trying to understand best ways of round tripping data.  Export from one collection and index into another collection.   Use our existing tooling as much as possible.
   
   # Solution
   
   Starting with a BATS test to demonstrate that bin/solr export and bin/solr post with .json file works.
   
   # Tests
   
   BATS.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2940:
URL: https://github.com/apache/solr/pull/2940#issuecomment-2587272197

   > Cool. Love the how-to nature of a BATS test here.
   
   I have a dream someday to re-organize all the Solr documentation along the lines of [Diátaxis](https://diataxis.fr/#).  I've used this model a few times and quite like it.   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #2940:
URL: https://github.com/apache/solr/pull/2940#discussion_r1911884285


##########
solr/solr-ref-guide/modules/deployment-guide/pages/solr-control-script-reference.adoc:
##########
@@ -1703,7 +1703,8 @@ The default is `\*:*` which will export all documents.
 |===
 +
 Comma separated list of fields to be exported.
-By default all fields are fetched.
+By default all fields are fetched.   
+If your schema uses `copyFields` and you are reindex the data, then you probably want to specify exactly which fields are being exported to simplify the import process.

Review Comment:
   ```suggestion
   If your schema uses `copyFields` and you reindex the data, then you probably want to specify exactly which fields are being exported to simplify the import process.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #2940:
URL: https://github.com/apache/solr/pull/2940#issuecomment-2582646623

   I'd love a plus one on the this before I merge...    For a lot of multi step processes, like exporting and importing data, I find modeling them as BATS tests makes them easier to understand in context.    I know that moves these BATS tests towards being integration or even system style tests...     I suspect that we may need to split the bats tests into system/integration/unit tests in the future..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #2940:
URL: https://github.com/apache/solr/pull/2940#discussion_r1915077934


##########
solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc:
##########
@@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data
 One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover.
 Of course this option can require more resources until the old collection can be deleted.
 
+=== Exporting/Importing Data from Solr
+
+Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back.

Review Comment:
   [-0] AFAIK `bin/solr export` uses the `/export` handler and [can only return fields that have docValues enabled](https://solr.apache.org/guide/solr/latest/query-guide/exporting-result-sets.html#field-requirements).
   
   That's a huge limitation that we should probably mention here!  Imagine someone's confusion when they follow these docs and somehow lose all of the text fields they were searching on!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2940: Demonstrate roundtrip export/import works
URL: https://github.com/apache/solr/pull/2940


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #2940:
URL: https://github.com/apache/solr/pull/2940#discussion_r1915089283


##########
solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc:
##########
@@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data
 One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover.
 Of course this option can require more resources until the old collection can be deleted.
 
+=== Exporting/Importing Data from Solr
+
+Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back.
+
+There are a number of third party tools that do this, see https://solr.cool/ for more information.   However, if you want to use what ships with Solr then we have some options:
+
+1. Use `bin/solr export` with the JSON output format (`.json`), and the `bin/solr post` tool to post that data back.

Review Comment:
   [Q] How would a user choose between these three options?  Or put differently - why would they choose one over the others?
   
   If there's no strong differentiator between the three, is there value in mentioning them all individually?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2940:
URL: https://github.com/apache/solr/pull/2940#issuecomment-3071374004

   This PR is now closed due to 60 days of inactivity after being marked as stale.  Re-opening this PR is still possible, in which case it will be marked as active again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2940:
URL: https://github.com/apache/solr/pull/2940#issuecomment-2727066623

   This PR has had no activity for 60 days and is now labeled as stale.  Any new activity will remove the stale label.  To attract more reviewers, please tag people who might be familiar with the code area and/or notify the dev@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org


Re: [PR] Demonstrate roundtrip export/import works [solr]

Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #2940:
URL: https://github.com/apache/solr/pull/2940#discussion_r1915077934


##########
solr/solr-ref-guide/modules/indexing-guide/pages/reindexing.adoc:
##########
@@ -165,6 +165,20 @@ Once the alias is in place and you are satisfied you no longer need the old data
 One advantage of this option is that you can switch back to the old collection if you discover problems our testing did not uncover.
 Of course this option can require more resources until the old collection can be deleted.
 
+=== Exporting/Importing Data from Solr
+
+Sometimes you don't want to run your full ETL pipeline to reindex into another collection, you just want to take the data in your existing collection, export it, and then import it back.

Review Comment:
   [-0] AFAIK `bin/solr export` uses the `/export` handler and [can only return fields that have docValues enabled](https://solr.apache.org/guide/solr/latest/query-guide/exporting-result-sets.html#field-requirements).  So it'd drop most text fields, etc.
   
   That's a huge limitation that we should probably mention here!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org