You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by tedsolr <ts...@sciquest.com> on 2014/10/31 15:44:08 UTC
exporting to CSV with solrj
I am trying to invoke the CSVResponseWriter to create a CSV file of all
stored fields. There are millions of documents so I need to write to the
file iteratively. I saw a snippet of code online that claimed it could
effectively remove the SorDocumentList wrapper and allow the docs to be
retrieved in the actual format requested in the query. However, I get a null
pointer from the CSVResponseWriter.write() method.
SolrQuery qry = new SolrQuery("*:*");
qry.setParam("wt", "csv");
// set other params
SolrServer server = getSolrServer();
try {
QueryResponse res = server.query(qry);
CSVResponseWriter writer = new CSVResponseWriter();
Writer w = new StringWriter();
SolrQueryResponse solrResponse = new SolrQueryResponse();
solrResponse.setAllValues(res.getResponse());
try {
SolrParams list = new MapSolrParams(new HashMap<String, String>());
writer.write(w, new LocalSolrQueryRequest(null, list), solrResponse);
} catch (IOException e) {
throw new RuntimeException(e);
}
System.out.print(w.toString());
} catch (SolrServerException e) {
e.printStackTrace();
}
NPE snippet:
org.apache.solr.response.CSVWriter.writeResponse(CSVResponseWriter.java:281)
org.apache.solr.response.CSVResponseWriter.write(CSVResponseWriter.java:56)
Am I on the right track with the approach? I really don't want to roll my
own document to CSV line convertor. Thanks!
Solr 4.9
--
View this message in context: http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: exporting to CSV with solrj
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
On 31 October 2014 14:58, will martin <wm...@gmail.com> wrote:
> "Why do you want to use CSV in SolrJ?" Alexandre are you looking for a
> design gig. This kind of question really begs nothing but disdain.
Nope. Not looking for a design gig. I give that advice away for free:
http://www.airpair.com/solr/workshops/discovering-your-inner-search-engine,
http://www.bigdatamontreal.org/?p=310 , http://www.solr-start.com/,
etc Though, in all fairness, I do charge for my Solr book:
https://www.packtpub.com/big-data-and-business-intelligence/instant-apache-solr-indexing-data-how-instant
In this particular case, there might have been two or three ways to
answer the question, depending on why Ted wanted to use CSV from SolrJ
as opposed to the more common command line approach, which is the
example given in the tutorial and online documentation. Depending on
his business-level goals, there might have been different types of
help offered. We, in the Solr community sometimes call it an X-Y
problem.
However, if you, Will Martin of USA, take a second-hand offence on
behalf of another person, I do apologize to you. There certainly was
no intent in upsetting innocent bystanders caught in the cross-fire of
determining the right answer to a slightly-unusual question.
Regards,
Alex.
Re: exporting to CSV with solrj
Posted by Erick Erickson <er...@gmail.com>.
@Will:
I can't tell you how many times questions like
"Why do you want to use CSV in SolrJ?" have
lead to solutions different from what the original
question might imply. It's a question I frequently
ask in almost the exact same way; it's a
perfectly legitimate question IMO.
Best,
Erick
On Fri, Oct 31, 2014 at 1:25 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : "Why do you want to use CSV in SolrJ?" Alexandre are you looking for a
>
> It's a legitmate question - part of providing good community support is
> making sure we understand *why* users are asking how to do something, so
> we can give good advice on other solutions people might not even have
> thought of -- teach a man to fish, vs give a man a fish, etc...
>
> https://people.apache.org/~hossman/#xyproblem
>
> ...if we understand *why* people ask questions, or aproach problems in
> certain ways, we can not only offer the best possible suggestions, but
> also consider how the underlying usecase (and other similar use cases like
> it) might be better served in the future.
>
> -Hoss
> http://www.lucidworks.com/
Re: exporting to CSV with solrj
Posted by Chris Hostetter <ho...@fucit.org>.
: "Why do you want to use CSV in SolrJ?" Alexandre are you looking for a
It's a legitmate question - part of providing good community support is
making sure we understand *why* users are asking how to do something, so
we can give good advice on other solutions people might not even have
thought of -- teach a man to fish, vs give a man a fish, etc...
https://people.apache.org/~hossman/#xyproblem
...if we understand *why* people ask questions, or aproach problems in
certain ways, we can not only offer the best possible suggestions, but
also consider how the underlying usecase (and other similar use cases like
it) might be better served in the future.
-Hoss
http://www.lucidworks.com/
Re: exporting to CSV with solrj
Posted by will martin <wm...@gmail.com>.
"Why do you want to use CSV in SolrJ?" Alexandre are you looking for a
design gig. This kind of question really begs nothing but disdain.
Commodity search exists, not matter what Paul Nelson writes and part of
that problem is due to advanced users always rewriting the reqs and specs
of less experienced users. <flame/>
On Fri, Oct 31, 2014 at 1:05 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> Why do you want to use CSV in SolrJ? You would just have to parse it again.
>
> You could just trigger that as a URL call from outside with cURL or as
> just an HTTP (not SolrJ) call from Java client.
>
> Regards,
> Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 31 October 2014 12:34, tedsolr <ts...@sciquest.com> wrote:
> > Sure thing, but how do I get the results output in CSV format?
> > response.getResults() is a list of SolrDocuments.
> >
> >
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845p4166861.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: exporting to CSV with solrj
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Why do you want to use CSV in SolrJ? You would just have to parse it again.
You could just trigger that as a URL call from outside with cURL or as
just an HTTP (not SolrJ) call from Java client.
Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
On 31 October 2014 12:34, tedsolr <ts...@sciquest.com> wrote:
> Sure thing, but how do I get the results output in CSV format?
> response.getResults() is a list of SolrDocuments.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845p4166861.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: exporting to CSV with solrj
Posted by tedsolr <ts...@sciquest.com>.
I think I'm getting the idea now. You either use the response writer via an
HTTP call, or you write your own exporter. Thanks to everyone for their
input.
--
View this message in context: http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845p4166889.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: exporting to CSV with solrj
Posted by Chris Hostetter <ho...@fucit.org>.
: Sure thing, but how do I get the results output in CSV format?
: response.getResults() is a list of SolrDocuments.
Either use something like the NoOpResponseParser which will give you the
entire response back as a single string, or implement your own
ResponseParser along hte lines of...
public class YourRawParser extends ResponseParser {
public NamedList<Object> processResponse(InputStream body, String encoding) {
// do whatever you want with the data in the InputStream
// as the data comes over the wire
doStuff(body);
// just ignore the result SolrServer gives you
return new NamedList<Object>();
}
}
-Hoss
http://www.lucidworks.com/
Re: exporting to CSV with solrj
Posted by tedsolr <ts...@sciquest.com>.
Sure thing, but how do I get the results output in CSV format?
response.getResults() is a list of SolrDocuments.
--
View this message in context: http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845p4166861.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: exporting to CSV with solrj
Posted by Jorge Luis Betancourt Gonzalez <jl...@uci.cu>.
When you fire a query against Solr with the wt=csv the response coming from Solr is *already* in CSV, the CSVResponseWriter is responsible for translating SolrDocument instances into a CSV on the server side, son I don’t see any reason on using it by your self, Solr already do the heavy lifting for you.
Regards,
On Oct 31, 2014, at 10:44 AM, tedsolr <ts...@sciquest.com> wrote:
> I am trying to invoke the CSVResponseWriter to create a CSV file of all
> stored fields. There are millions of documents so I need to write to the
> file iteratively. I saw a snippet of code online that claimed it could
> effectively remove the SorDocumentList wrapper and allow the docs to be
> retrieved in the actual format requested in the query. However, I get a null
> pointer from the CSVResponseWriter.write() method.
>
> SolrQuery qry = new SolrQuery("*:*");
> qry.setParam("wt", "csv");
> // set other params
> SolrServer server = getSolrServer();
> try {
> QueryResponse res = server.query(qry);
>
> CSVResponseWriter writer = new CSVResponseWriter();
> Writer w = new StringWriter();
> SolrQueryResponse solrResponse = new SolrQueryResponse();
> solrResponse.setAllValues(res.getResponse());
> try {
> SolrParams list = new MapSolrParams(new HashMap<String, String>());
> writer.write(w, new LocalSolrQueryRequest(null, list), solrResponse);
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> System.out.print(w.toString());
>
> } catch (SolrServerException e) {
> e.printStackTrace();
> }
>
> NPE snippet:
> org.apache.solr.response.CSVWriter.writeResponse(CSVResponseWriter.java:281)
> org.apache.solr.response.CSVResponseWriter.write(CSVResponseWriter.java:56)
>
> Am I on the right track with the approach? I really don't want to roll my
> own document to CSV line convertor. Thanks!
> Solr 4.9
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/exporting-to-CSV-with-solrj-tp4166845.html
> Sent from the Solr - User mailing list archive at Nabble.com.