You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Helmut Hoffer von Ankershoffen <he...@googlemail.com> on 2011/06/09 21:31:41 UTC
Processing/Indexing CSV
Hi,
there seems to be no way to index CSV using the DataImportHandler.
Using a combination of
LineEntityProcessor<http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
and RegexTransformer<http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
as
proposed in
http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
not working for real world CSV files.
E.g. many CSV files have double-quotes enclosing some but not all columns -
there is no elegant way to segment this using a simple regular expression.
As CSV is still very common esp. in E-Commerce scenarios, I propose that
Solr provides a CSVEntityProcessor that:
1) Handles the case of CSV files with/without and with some double-quote
enclosed columns
2) Allows for a configurable column separator (';',',','\t' etc.)
3) Allows for a leading row containing column headings
4) If there is a leading row with column headings provides a possibility to
address columns by their column names and map them to Solr fields (similar
to the XPathEntityProcessor)
5) Auto-detects encoding of the file (UTF-8 etc.)
This would make it A LOT easier to use Solr for E-Commerce scenarios.
If there is no such entity processor in the works i will develop one ... So
please let me know.
Regards
Re: Processing/Indexing CSV
Posted by Ken Krugler <kk...@transpac.com>.
On Jun 9, 2011, at 2:21pm, Helmut Hoffer von Ankershoffen wrote:
> Hi,
>
> btw: there seems to somewhat of a non-match regarding efforts to Enhance DIH
> regarding the CSV format (James Dyer) and the effort to maintain the
> CSVLoader (Ken Krugler). How about merging your efforts and migrating the
> CSVLoader to a CSVEntityProcessor (cp. my initial email)? :-)
While I'm a CSVLoader user (and I've found/fixed one bug in it), I'm not involved in any active development/maintenance of that piece of code.
If James or you can make progress on merging support for CSV into DIH, that's great.
-- Ken
> On Thu, Jun 9, 2011 at 11:17 PM, Helmut Hoffer von Ankershoffen <
> helmuthva@googlemail.com> wrote:
>
>>
>>
>> On Thu, Jun 9, 2011 at 11:05 PM, Ken Krugler <kk...@transpac.com>wrote:
>>
>>>
>>> On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
>>>
>>>> Hi,
>>>>
>>>> ... that would be an option if there is a defined set of field names and
>>> a
>>>> single column/CSV layout. The scenario however is different csv files
>>> (from
>>>> different shops) with individual column layouts (separators, encodings
>>>> etc.). The idea is to map known field names to defined field names in
>>> the
>>>> solr schema. If I understand the capabilities of the CSVLoader correctly
>>>> (sorry, I am completely new to Solr, started work on it today) this is
>>> not
>>>> possible - is it?
>>>
>>> As per the documentation on
>>> http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the
>>> names/positions of fields in the CSV file, and ignore fieldnames.
>>>
>>> So this seems like it would solve your requirement, as each different
>>> layout could specify its own such mapping during import.
>>>
>>> Sure, but the requirement (to keep the process of integrating new shops
>> efficient) is not to have one mapping per import (cp. the Email regarding
>> "more or less schema free") but to enhance one mapping that maps common
>> field names to defined fields disregarding order of known fields/columns. As
>> far as I understand that is not a problem at all with DIH, however DIH and
>> CSV are not a perfect match ,-)
>>
>>
>>> It could be handy to provide a fieldname map (versus the value map that
>>> UpdateCSV supports).
>>
>> Definitely. Either a fieldname map in CSVLoader or a robust CSVLoader in
>> DIH ...
>>
>>
>>> Then you could use the header, and just provide a mapping from header
>>> fieldnames to schema fieldnames.
>>>
>> That's the idea -)
>>
>> => what's the best way to progress. Either someone enhances the CSVLoader
>> by a field mapper (with multipel input field names mapping to one field name
>> in the Solr schema) or someone enhances the DIH with a robust CSV loader
>> ,-). As I am completely new to this Community, please give me the direction
>> to go (or wait :-).
>>
>> best regards
>>
>>
>>> -- Ken
>>>
>>>> On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <
>>> yonik@lucidimagination.com>wrote:
>>>>
>>>>> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
>>>>> <he...@googlemail.com> wrote:
>>>>>> Hi,
>>>>>> yes, it's about CSV files loaded via HTTP from shops to be fed into a
>>>>>> shopping search engine.
>>>>>> The CSV Loader cannot map fields (only field values) etc.
>>>>>
>>>>> You can provide your own list of fieldnames and optionally ignore the
>>>>> first line of the CSV file (assuming it contains the field names).
>>>>> http://wiki.apache.org/solr/UpdateCSV#fieldnames
>>>>>
>>>>> -Yonik
>>>>> http://www.lucidimagination.com
>>>>>
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://bixolabs.com
>>> custom data mining solutions
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
btw: there seems to somewhat of a non-match regarding efforts to Enhance DIH
regarding the CSV format (James Dyer) and the effort to maintain the
CSVLoader (Ken Krugler). How about merging your efforts and migrating the
CSVLoader to a CSVEntityProcessor (cp. my initial email)? :-)
Best Regards
On Thu, Jun 9, 2011 at 11:17 PM, Helmut Hoffer von Ankershoffen <
helmuthva@googlemail.com> wrote:
>
>
> On Thu, Jun 9, 2011 at 11:05 PM, Ken Krugler <kk...@transpac.com>wrote:
>
>>
>> On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
>>
>> > Hi,
>> >
>> > ... that would be an option if there is a defined set of field names and
>> a
>> > single column/CSV layout. The scenario however is different csv files
>> (from
>> > different shops) with individual column layouts (separators, encodings
>> > etc.). The idea is to map known field names to defined field names in
>> the
>> > solr schema. If I understand the capabilities of the CSVLoader correctly
>> > (sorry, I am completely new to Solr, started work on it today) this is
>> not
>> > possible - is it?
>>
>> As per the documentation on
>> http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the
>> names/positions of fields in the CSV file, and ignore fieldnames.
>>
>> So this seems like it would solve your requirement, as each different
>> layout could specify its own such mapping during import.
>>
>> Sure, but the requirement (to keep the process of integrating new shops
> efficient) is not to have one mapping per import (cp. the Email regarding
> "more or less schema free") but to enhance one mapping that maps common
> field names to defined fields disregarding order of known fields/columns. As
> far as I understand that is not a problem at all with DIH, however DIH and
> CSV are not a perfect match ,-)
>
>
>> It could be handy to provide a fieldname map (versus the value map that
>> UpdateCSV supports).
>
> Definitely. Either a fieldname map in CSVLoader or a robust CSVLoader in
> DIH ...
>
>
>> Then you could use the header, and just provide a mapping from header
>> fieldnames to schema fieldnames.
>>
> That's the idea -)
>
> => what's the best way to progress. Either someone enhances the CSVLoader
> by a field mapper (with multipel input field names mapping to one field name
> in the Solr schema) or someone enhances the DIH with a robust CSV loader
> ,-). As I am completely new to this Community, please give me the direction
> to go (or wait :-).
>
> best regards
>
>
>> -- Ken
>>
>> > On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <
>> yonik@lucidimagination.com>wrote:
>> >
>> >> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
>> >> <he...@googlemail.com> wrote:
>> >>> Hi,
>> >>> yes, it's about CSV files loaded via HTTP from shops to be fed into a
>> >>> shopping search engine.
>> >>> The CSV Loader cannot map fields (only field values) etc.
>> >>
>> >> You can provide your own list of fieldnames and optionally ignore the
>> >> first line of the CSV file (assuming it contains the field names).
>> >> http://wiki.apache.org/solr/UpdateCSV#fieldnames
>> >>
>> >> -Yonik
>> >> http://www.lucidimagination.com
>> >>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> custom data mining solutions
>>
>>
>>
>>
>>
>>
>>
>
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
thanks for the Intro, will do next week :-)
greetings from berlin
On Fri, Jun 10, 2011 at 2:49 PM, Erick Erickson <er...@gmail.com>wrote:
> Well, here's a place to start if you want to patch the code:
>
> http://wiki.apache.org/solr/HowToContribute
>
> If you do want to take this on, hop on over to the dev list
> and start a discussion. I'd start with some posts on that list
> before entering or working on a JIRA issue, just ask for
> some guidance. A good place to start is pretty much what
> you've done here, state your problem, and what you think
> the correct behavior is.
>
> Be prepared for things to be brought up you never thought
> of <G>... which is the point of starting the discussion there.
>
> A very good way to start is to get the code, compile it, and then
> run some of the test cases in an IDE, stepping through the test
> case in the debugger. Sometimes that doesn't work easily, but
> if it does it gives you an idea of how the code works. There are
> instructions at the above link for setting things up in an IDE
> (Eclipse and Intellij are popular).
>
> Just loading the project and looking for files that begin with
> CSV might be a place to start. Then look for files that begin
> with TestCSV. Both of these "look promising".
>
> Anyway, if you get that far, then go over to the dev list and say
> "I'm thinking of XXX, this code appears to be handled in YYY and
> I'm thinking of changing it like ZZZ" and it will be well received.
>
> Of course if you want to go ahead and make your changes and submit
> a patch, that's even better, but it's often best to get a bit of guidance
> first.
>
> Best
> Erick
>
> On Thu, Jun 9, 2011 at 5:17 PM, Helmut Hoffer von Ankershoffen
> <he...@googlemail.com> wrote:
> > On Thu, Jun 9, 2011 at 11:05 PM, Ken Krugler <
> kkrugler_lists@transpac.com>wrote:
> >
> >>
> >> On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
> >>
> >> > Hi,
> >> >
> >> > ... that would be an option if there is a defined set of field names
> and
> >> a
> >> > single column/CSV layout. The scenario however is different csv files
> >> (from
> >> > different shops) with individual column layouts (separators, encodings
> >> > etc.). The idea is to map known field names to defined field names in
> the
> >> > solr schema. If I understand the capabilities of the CSVLoader
> correctly
> >> > (sorry, I am completely new to Solr, started work on it today) this is
> >> not
> >> > possible - is it?
> >>
> >> As per the documentation on
> >> http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the
> >> names/positions of fields in the CSV file, and ignore fieldnames.
> >>
> >> So this seems like it would solve your requirement, as each different
> >> layout could specify its own such mapping during import.
> >>
> >> Sure, but the requirement (to keep the process of integrating new shops
> > efficient) is not to have one mapping per import (cp. the Email regarding
> > "more or less schema free") but to enhance one mapping that maps common
> > field names to defined fields disregarding order of known fields/columns.
> As
> > far as I understand that is not a problem at all with DIH, however DIH
> and
> > CSV are not a perfect match ,-)
> >
> >
> >> It could be handy to provide a fieldname map (versus the value map that
> >> UpdateCSV supports).
> >
> > Definitely. Either a fieldname map in CSVLoader or a robust CSVLoader in
> DIH
> > ...
> >
> >
> >> Then you could use the header, and just provide a mapping from header
> >> fieldnames to schema fieldnames.
> >>
> > That's the idea -)
> >
> > => what's the best way to progress. Either someone enhances the CSVLoader
> by
> > a field mapper (with multipel input field names mapping to one field name
> in
> > the Solr schema) or someone enhances the DIH with a robust CSV loader
> ,-).
> > As I am completely new to this Community, please give me the direction to
> go
> > (or wait :-).
> >
> > best regards
> >
> >
> >> -- Ken
> >>
> >> > On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <
> >> yonik@lucidimagination.com>wrote:
> >> >
> >> >> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
> >> >> <he...@googlemail.com> wrote:
> >> >>> Hi,
> >> >>> yes, it's about CSV files loaded via HTTP from shops to be fed into
> a
> >> >>> shopping search engine.
> >> >>> The CSV Loader cannot map fields (only field values) etc.
> >> >>
> >> >> You can provide your own list of fieldnames and optionally ignore the
> >> >> first line of the CSV file (assuming it contains the field names).
> >> >> http://wiki.apache.org/solr/UpdateCSV#fieldnames
> >> >>
> >> >> -Yonik
> >> >> http://www.lucidimagination.com
> >> >>
> >>
> >> --------------------------
> >> Ken Krugler
> >> +1 530-210-6378
> >> http://bixolabs.com
> >> custom data mining solutions
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
>
Re: Processing/Indexing CSV
Posted by Erick Erickson <er...@gmail.com>.
Well, here's a place to start if you want to patch the code:
http://wiki.apache.org/solr/HowToContribute
If you do want to take this on, hop on over to the dev list
and start a discussion. I'd start with some posts on that list
before entering or working on a JIRA issue, just ask for
some guidance. A good place to start is pretty much what
you've done here, state your problem, and what you think
the correct behavior is.
Be prepared for things to be brought up you never thought
of <G>... which is the point of starting the discussion there.
A very good way to start is to get the code, compile it, and then
run some of the test cases in an IDE, stepping through the test
case in the debugger. Sometimes that doesn't work easily, but
if it does it gives you an idea of how the code works. There are
instructions at the above link for setting things up in an IDE
(Eclipse and Intellij are popular).
Just loading the project and looking for files that begin with
CSV might be a place to start. Then look for files that begin
with TestCSV. Both of these "look promising".
Anyway, if you get that far, then go over to the dev list and say
"I'm thinking of XXX, this code appears to be handled in YYY and
I'm thinking of changing it like ZZZ" and it will be well received.
Of course if you want to go ahead and make your changes and submit
a patch, that's even better, but it's often best to get a bit of guidance first.
Best
Erick
On Thu, Jun 9, 2011 at 5:17 PM, Helmut Hoffer von Ankershoffen
<he...@googlemail.com> wrote:
> On Thu, Jun 9, 2011 at 11:05 PM, Ken Krugler <kk...@transpac.com>wrote:
>
>>
>> On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
>>
>> > Hi,
>> >
>> > ... that would be an option if there is a defined set of field names and
>> a
>> > single column/CSV layout. The scenario however is different csv files
>> (from
>> > different shops) with individual column layouts (separators, encodings
>> > etc.). The idea is to map known field names to defined field names in the
>> > solr schema. If I understand the capabilities of the CSVLoader correctly
>> > (sorry, I am completely new to Solr, started work on it today) this is
>> not
>> > possible - is it?
>>
>> As per the documentation on
>> http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the
>> names/positions of fields in the CSV file, and ignore fieldnames.
>>
>> So this seems like it would solve your requirement, as each different
>> layout could specify its own such mapping during import.
>>
>> Sure, but the requirement (to keep the process of integrating new shops
> efficient) is not to have one mapping per import (cp. the Email regarding
> "more or less schema free") but to enhance one mapping that maps common
> field names to defined fields disregarding order of known fields/columns. As
> far as I understand that is not a problem at all with DIH, however DIH and
> CSV are not a perfect match ,-)
>
>
>> It could be handy to provide a fieldname map (versus the value map that
>> UpdateCSV supports).
>
> Definitely. Either a fieldname map in CSVLoader or a robust CSVLoader in DIH
> ...
>
>
>> Then you could use the header, and just provide a mapping from header
>> fieldnames to schema fieldnames.
>>
> That's the idea -)
>
> => what's the best way to progress. Either someone enhances the CSVLoader by
> a field mapper (with multipel input field names mapping to one field name in
> the Solr schema) or someone enhances the DIH with a robust CSV loader ,-).
> As I am completely new to this Community, please give me the direction to go
> (or wait :-).
>
> best regards
>
>
>> -- Ken
>>
>> > On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <
>> yonik@lucidimagination.com>wrote:
>> >
>> >> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
>> >> <he...@googlemail.com> wrote:
>> >>> Hi,
>> >>> yes, it's about CSV files loaded via HTTP from shops to be fed into a
>> >>> shopping search engine.
>> >>> The CSV Loader cannot map fields (only field values) etc.
>> >>
>> >> You can provide your own list of fieldnames and optionally ignore the
>> >> first line of the CSV file (assuming it contains the field names).
>> >> http://wiki.apache.org/solr/UpdateCSV#fieldnames
>> >>
>> >> -Yonik
>> >> http://www.lucidimagination.com
>> >>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> custom data mining solutions
>>
>>
>>
>>
>>
>>
>>
>
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
On Thu, Jun 9, 2011 at 11:05 PM, Ken Krugler <kk...@transpac.com>wrote:
>
> On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
>
> > Hi,
> >
> > ... that would be an option if there is a defined set of field names and
> a
> > single column/CSV layout. The scenario however is different csv files
> (from
> > different shops) with individual column layouts (separators, encodings
> > etc.). The idea is to map known field names to defined field names in the
> > solr schema. If I understand the capabilities of the CSVLoader correctly
> > (sorry, I am completely new to Solr, started work on it today) this is
> not
> > possible - is it?
>
> As per the documentation on
> http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the
> names/positions of fields in the CSV file, and ignore fieldnames.
>
> So this seems like it would solve your requirement, as each different
> layout could specify its own such mapping during import.
>
> Sure, but the requirement (to keep the process of integrating new shops
efficient) is not to have one mapping per import (cp. the Email regarding
"more or less schema free") but to enhance one mapping that maps common
field names to defined fields disregarding order of known fields/columns. As
far as I understand that is not a problem at all with DIH, however DIH and
CSV are not a perfect match ,-)
> It could be handy to provide a fieldname map (versus the value map that
> UpdateCSV supports).
Definitely. Either a fieldname map in CSVLoader or a robust CSVLoader in DIH
...
> Then you could use the header, and just provide a mapping from header
> fieldnames to schema fieldnames.
>
That's the idea -)
=> what's the best way to progress. Either someone enhances the CSVLoader by
a field mapper (with multipel input field names mapping to one field name in
the Solr schema) or someone enhances the DIH with a robust CSV loader ,-).
As I am completely new to this Community, please give me the direction to go
(or wait :-).
best regards
> -- Ken
>
> > On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <
> yonik@lucidimagination.com>wrote:
> >
> >> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
> >> <he...@googlemail.com> wrote:
> >>> Hi,
> >>> yes, it's about CSV files loaded via HTTP from shops to be fed into a
> >>> shopping search engine.
> >>> The CSV Loader cannot map fields (only field values) etc.
> >>
> >> You can provide your own list of fieldnames and optionally ignore the
> >> first line of the CSV file (assuming it contains the field names).
> >> http://wiki.apache.org/solr/UpdateCSV#fieldnames
> >>
> >> -Yonik
> >> http://www.lucidimagination.com
> >>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> custom data mining solutions
>
>
>
>
>
>
>
Re: Processing/Indexing CSV
Posted by Ken Krugler <kk...@transpac.com>.
On Jun 9, 2011, at 1:27pm, Helmut Hoffer von Ankershoffen wrote:
> Hi,
>
> ... that would be an option if there is a defined set of field names and a
> single column/CSV layout. The scenario however is different csv files (from
> different shops) with individual column layouts (separators, encodings
> etc.). The idea is to map known field names to defined field names in the
> solr schema. If I understand the capabilities of the CSVLoader correctly
> (sorry, I am completely new to Solr, started work on it today) this is not
> possible - is it?
As per the documentation on http://wiki.apache.org/solr/UpdateCSV#fieldnames, you can specify the names/positions of fields in the CSV file, and ignore fieldnames.
So this seems like it would solve your requirement, as each different layout could specify its own such mapping during import.
It could be handy to provide a fieldname map (versus the value map that UpdateCSV supports). Then you could use the header, and just provide a mapping from header fieldnames to schema fieldnames.
-- Ken
> On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
>
>> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
>> <he...@googlemail.com> wrote:
>>> Hi,
>>> yes, it's about CSV files loaded via HTTP from shops to be fed into a
>>> shopping search engine.
>>> The CSV Loader cannot map fields (only field values) etc.
>>
>> You can provide your own list of fieldnames and optionally ignore the
>> first line of the CSV file (assuming it contains the field names).
>> http://wiki.apache.org/solr/UpdateCSV#fieldnames
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
... that would be an option if there is a defined set of field names and a
single column/CSV layout. The scenario however is different csv files (from
different shops) with individual column layouts (separators, encodings
etc.). The idea is to map known field names to defined field names in the
solr schema. If I understand the capabilities of the CSVLoader correctly
(sorry, I am completely new to Solr, started work on it today) this is not
possible - is it?
Best Regards
On Thu, Jun 9, 2011 at 10:12 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
> <he...@googlemail.com> wrote:
> > Hi,
> > yes, it's about CSV files loaded via HTTP from shops to be fed into a
> > shopping search engine.
> > The CSV Loader cannot map fields (only field values) etc.
>
> You can provide your own list of fieldnames and optionally ignore the
> first line of the CSV file (assuming it contains the field names).
> http://wiki.apache.org/solr/UpdateCSV#fieldnames
>
> -Yonik
> http://www.lucidimagination.com
>
Re: Processing/Indexing CSV
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jun 9, 2011 at 4:07 PM, Helmut Hoffer von Ankershoffen
<he...@googlemail.com> wrote:
> Hi,
> yes, it's about CSV files loaded via HTTP from shops to be fed into a
> shopping search engine.
> The CSV Loader cannot map fields (only field values) etc.
You can provide your own list of fieldnames and optionally ignore the
first line of the CSV file (assuming it contains the field names).
http://wiki.apache.org/solr/UpdateCSV#fieldnames
-Yonik
http://www.lucidimagination.com
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
yes, it's about CSV files loaded via HTTP from shops to be fed into a
shopping search engine.
The CSV Loader cannot map fields (only field values) etc. DIH is flexible
enough for building the importing part of such a thing but misses elegant
handling of CSV data ...
Regards
On Thu, Jun 9, 2011 at 9:50 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Thu, Jun 9, 2011 at 3:31 PM, Helmut Hoffer von Ankershoffen
> <he...@googlemail.com> wrote:
> > Hi,
> >
> > there seems to be no way to index CSV using the DataImportHandler.
>
> Looking over the features you want, it looks like you're starting from
> a CSV file (as opposed to CSV stored in a database).
> Is there a reason that you need to use DIH and can't directly use the
> CSV loader?
> http://wiki.apache.org/solr/UpdateCSV
>
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> > Using a combination of
> > LineEntityProcessor<
> http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
> > and RegexTransformer<
> http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
> > as
> > proposed in
> >
> http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
> > not working for real world CSV files.
> >
> > E.g. many CSV files have double-quotes enclosing some but not all columns
> -
> > there is no elegant way to segment this using a simple regular
> expression.
> >
> > As CSV is still very common esp. in E-Commerce scenarios, I propose that
> > Solr provides a CSVEntityProcessor that:
> > 1) Handles the case of CSV files with/without and with some double-quote
> > enclosed columns
> > 2) Allows for a configurable column separator (';',',','\t' etc.)
> > 3) Allows for a leading row containing column headings
> > 4) If there is a leading row with column headings provides a possibility
> to
> > address columns by their column names and map them to Solr fields
> (similar
> > to the XPathEntityProcessor)
> > 5) Auto-detects encoding of the file (UTF-8 etc.)
> >
> > This would make it A LOT easier to use Solr for E-Commerce scenarios.
> >
> > If there is no such entity processor in the works i will develop one ...
> So
> > please let me know.
> >
> > Regards
> >
>
Re: Processing/Indexing CSV
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jun 9, 2011 at 3:31 PM, Helmut Hoffer von Ankershoffen
<he...@googlemail.com> wrote:
> Hi,
>
> there seems to be no way to index CSV using the DataImportHandler.
Looking over the features you want, it looks like you're starting from
a CSV file (as opposed to CSV stored in a database).
Is there a reason that you need to use DIH and can't directly use the
CSV loader?
http://wiki.apache.org/solr/UpdateCSV
-Yonik
http://www.lucidimagination.com
> Using a combination of
> LineEntityProcessor<http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
> and RegexTransformer<http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
> as
> proposed in
> http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
> not working for real world CSV files.
>
> E.g. many CSV files have double-quotes enclosing some but not all columns -
> there is no elegant way to segment this using a simple regular expression.
>
> As CSV is still very common esp. in E-Commerce scenarios, I propose that
> Solr provides a CSVEntityProcessor that:
> 1) Handles the case of CSV files with/without and with some double-quote
> enclosed columns
> 2) Allows for a configurable column separator (';',',','\t' etc.)
> 3) Allows for a leading row containing column headings
> 4) If there is a leading row with column headings provides a possibility to
> address columns by their column names and map them to Solr fields (similar
> to the XPathEntityProcessor)
> 5) Auto-detects encoding of the file (UTF-8 etc.)
>
> This would make it A LOT easier to use Solr for E-Commerce scenarios.
>
> If there is no such entity processor in the works i will develop one ... So
> please let me know.
>
> Regards
>
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
s/provide and/provide any/ig ,-)
On Thu, Jun 9, 2011 at 10:01 PM, Helmut Hoffer von Ankershoffen <
helmuthva@googlemail.com> wrote:
> Hi,
>
> just looked at your code. Definitely an improvement :-)
>
> The problem with the double-quotes is, that the delimiter (let's say ',')
> might be part of the column value. The goal is to process something like
> this without any tricky configuration
>
> name1,name2,name3
> val1,"val2,...",val3
> ...
>
> The user should not have to provide and before-hand knowledge regarding the
> column layout or the encoding of the CSV file. Ideally the only thing that
> has to be specified is firstLineHasFieldnames="true" separator=";".
> Autodetecting the separator and encoding would be even more elegant.
>
> If nobody else has this in the works I will start building such a patch
> next week.
>
> Best Regards
>
>
> On Thu, Jun 9, 2011 at 9:45 PM, Dyer, James <Ja...@ingrambook.com>wrote:
>
>> Helmut,
>>
>> I recently submitted SOLR-2549 (
>> https://issues.apache.org/jira/browse/SOLR-2549) to handle both
>> fixed-width and delimited flat files. To be honest, I only needed
>> fixed-width support for my app so this might not support everything you
>> mention for delimited files, but it should be a good start.
>>
>> In particular, you might need to enhance this to handle the double quotes
>> (I had though a delimiter regex along these lines might handle it:
>> (?:[\"]?[,]|[\"]$) ... note this is a sample I just cooked up quick and no
>> doubt has errors, and maybe as you say a simple regex might not work at all
>> ) ... I also didn't do anything with encodings but I'm not sure this will be
>> an issue either...
>>
>> James Dyer
>> E-Commerce Systems
>> Ingram Content Group
>> (615) 213-4311
>>
>> -----Original Message-----
>> From: Helmut Hoffer von Ankershoffen [mailto:helmuthva@googlemail.com]
>> Sent: Thursday, June 09, 2011 2:32 PM
>> To: solr-user@lucene.apache.org
>> Subject: Processing/Indexing CSV
>>
>> Hi,
>>
>> there seems to be no way to index CSV using the DataImportHandler.
>>
>> Using a combination of
>> LineEntityProcessor<
>> http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
>> and RegexTransformer<
>> http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
>> as
>> proposed in
>>
>> http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
>> not working for real world CSV files.
>>
>> E.g. many CSV files have double-quotes enclosing some but not all columns
>> -
>> there is no elegant way to segment this using a simple regular expression.
>>
>> As CSV is still very common esp. in E-Commerce scenarios, I propose that
>> Solr provides a CSVEntityProcessor that:
>> 1) Handles the case of CSV files with/without and with some double-quote
>> enclosed columns
>> 2) Allows for a configurable column separator (';',',','\t' etc.)
>> 3) Allows for a leading row containing column headings
>> 4) If there is a leading row with column headings provides a possibility
>> to
>> address columns by their column names and map them to Solr fields (similar
>> to the XPathEntityProcessor)
>> 5) Auto-detects encoding of the file (UTF-8 etc.)
>>
>> This would make it A LOT easier to use Solr for E-Commerce scenarios.
>>
>> If there is no such entity processor in the works i will develop one ...
>> So
>> please let me know.
>>
>> Regards
>>
>
>
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
just looked at your code. Definitely an improvement :-)
The problem with the double-quotes is, that the delimiter (let's say ',')
might be part of the column value. The goal is to process something like
this without any tricky configuration
name1,name2,name3
val1,"val2,...",val3
...
The user should not have to provide and before-hand knowledge regarding the
column layout or the encoding of the CSV file. Ideally the only thing that
has to be specified is firstLineHasFieldnames="true" separator=";".
Autodetecting the separator and encoding would be even more elegant.
If nobody else has this in the works I will start building such a patch next
week.
Best Regards
On Thu, Jun 9, 2011 at 9:45 PM, Dyer, James <Ja...@ingrambook.com>wrote:
> Helmut,
>
> I recently submitted SOLR-2549 (
> https://issues.apache.org/jira/browse/SOLR-2549) to handle both
> fixed-width and delimited flat files. To be honest, I only needed
> fixed-width support for my app so this might not support everything you
> mention for delimited files, but it should be a good start.
>
> In particular, you might need to enhance this to handle the double quotes
> (I had though a delimiter regex along these lines might handle it:
> (?:[\"]?[,]|[\"]$) ... note this is a sample I just cooked up quick and no
> doubt has errors, and maybe as you say a simple regex might not work at all
> ) ... I also didn't do anything with encodings but I'm not sure this will be
> an issue either...
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
> -----Original Message-----
> From: Helmut Hoffer von Ankershoffen [mailto:helmuthva@googlemail.com]
> Sent: Thursday, June 09, 2011 2:32 PM
> To: solr-user@lucene.apache.org
> Subject: Processing/Indexing CSV
>
> Hi,
>
> there seems to be no way to index CSV using the DataImportHandler.
>
> Using a combination of
> LineEntityProcessor<
> http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
> and RegexTransformer<
> http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
> as
> proposed in
>
> http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
> not working for real world CSV files.
>
> E.g. many CSV files have double-quotes enclosing some but not all columns -
> there is no elegant way to segment this using a simple regular expression.
>
> As CSV is still very common esp. in E-Commerce scenarios, I propose that
> Solr provides a CSVEntityProcessor that:
> 1) Handles the case of CSV files with/without and with some double-quote
> enclosed columns
> 2) Allows for a configurable column separator (';',',','\t' etc.)
> 3) Allows for a leading row containing column headings
> 4) If there is a leading row with column headings provides a possibility to
> address columns by their column names and map them to Solr fields (similar
> to the XPathEntityProcessor)
> 5) Auto-detects encoding of the file (UTF-8 etc.)
>
> This would make it A LOT easier to use Solr for E-Commerce scenarios.
>
> If there is no such entity processor in the works i will develop one ... So
> please let me know.
>
> Regards
>
RE: Processing/Indexing CSV
Posted by "Dyer, James" <Ja...@ingrambook.com>.
Helmut,
I recently submitted SOLR-2549 (https://issues.apache.org/jira/browse/SOLR-2549) to handle both fixed-width and delimited flat files. To be honest, I only needed fixed-width support for my app so this might not support everything you mention for delimited files, but it should be a good start.
In particular, you might need to enhance this to handle the double quotes (I had though a delimiter regex along these lines might handle it: (?:[\"]?[,]|[\"]$) ... note this is a sample I just cooked up quick and no doubt has errors, and maybe as you say a simple regex might not work at all ) ... I also didn't do anything with encodings but I'm not sure this will be an issue either...
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Helmut Hoffer von Ankershoffen [mailto:helmuthva@googlemail.com]
Sent: Thursday, June 09, 2011 2:32 PM
To: solr-user@lucene.apache.org
Subject: Processing/Indexing CSV
Hi,
there seems to be no way to index CSV using the DataImportHandler.
Using a combination of
LineEntityProcessor<http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
and RegexTransformer<http://wiki.apache.org/solr/DataImportHandler#RegexTransformer>
as
proposed in
http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is
not working for real world CSV files.
E.g. many CSV files have double-quotes enclosing some but not all columns -
there is no elegant way to segment this using a simple regular expression.
As CSV is still very common esp. in E-Commerce scenarios, I propose that
Solr provides a CSVEntityProcessor that:
1) Handles the case of CSV files with/without and with some double-quote
enclosed columns
2) Allows for a configurable column separator (';',',','\t' etc.)
3) Allows for a leading row containing column headings
4) If there is a leading row with column headings provides a possibility to
address columns by their column names and map them to Solr fields (similar
to the XPathEntityProcessor)
5) Auto-detects encoding of the file (UTF-8 etc.)
This would make it A LOT easier to use Solr for E-Commerce scenarios.
If there is no such entity processor in the works i will develop one ... So
please let me know.
Regards
Re: Processing/Indexing CSV
Posted by Helmut Hoffer von Ankershoffen <he...@googlemail.com>.
Hi,
to make my point more clear: if the CSV has a fixed schema / column layout,
using the RegexTransformer is of course a possibility (however awkward). But
if you want to implement a (more or less) schema free shopping search engine
...
regards
On Thu, Jun 9, 2011 at 9:31 PM, Helmut Hoffer von Ankershoffen <
helmuthva@googlemail.com> wrote:
> Hi,
>
> there seems to be no way to index CSV using the DataImportHandler.
>
> Using a combination of LineEntityProcessor<http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor>
> and RegexTransformer<http://wiki.apache.org/solr/DataImportHandler#RegexTransformer> as
> proposed in
> http://robotlibrarian.billdueber.com/an-exercise-in-solr-and-dataimporthandler-hathitrust-data/is not working for real world CSV files.
>
> E.g. many CSV files have double-quotes enclosing some but not all columns -
> there is no elegant way to segment this using a simple regular expression.
>
> As CSV is still very common esp. in E-Commerce scenarios, I propose that
> Solr provides a CSVEntityProcessor that:
> 1) Handles the case of CSV files with/without and with some double-quote
> enclosed columns
> 2) Allows for a configurable column separator (';',',','\t' etc.)
> 3) Allows for a leading row containing column headings
> 4) If there is a leading row with column headings provides a possibility to
> address columns by their column names and map them to Solr fields (similar
> to the XPathEntityProcessor)
> 5) Auto-detects encoding of the file (UTF-8 etc.)
>
> This would make it A LOT easier to use Solr for E-Commerce scenarios.
>
> If there is no such entity processor in the works i will develop one ... So
> please let me know.
>
> Regards
>