You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Karl Stoney <Ka...@autotrader.co.uk.INVALID> on 2021/03/22 13:00:51 UTC
REINDEXCOLLECTION unknown field
Hi,
Sorry for all the questions recently…
So as per https://solr.apache.org/guide/8_0/reindexing.html; we’re trying to remove a load of fields. Subsequently we’ve created a new collection with the new schema and we’re attempting to reindex from old to new.
There’s about 216 fields in total being removed…
The REINDEX fails though, because the field has been removed:
2:52:31.508 [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in DaemonStream: at-uk-003
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr-0.search-solr.svc.cluster.local:80/solr/at-uk-003: ERROR: [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field 'OEM_ENGINE_MARKETING'
Surely this field should simply just be ignored?
I can’t see any way of using REINDEXCOLLECTION to reindex data from one collection to another where we have removed fields.
Any input would be appreciated.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Re: REINDEXCOLLECTION unknown field
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
There may also be a way to drop a bunch of fields on intake by crafting a
custom update request processor chain in solrconfig.xml.
Or by temporarily declaring them with stored=false, indexed=false in the
target schema.
As long as nothing actually ends up in Lucene segments, you can change
schema definition later.
Regards,
Alex
On Mon., Mar. 22, 2021, 12:25 p.m. Karl Stoney,
<Ka...@autotrader.co.uk.invalid> wrote:
> Interestingly enough the next issue we hit is the `fl=750 fields` == too
> big, so we switched to using POST/x-url-form-encoded for REINDEXCOLLECTION
> which accepts the request, but then it silently fails (no logs in solr,
> just doesn't work).
> I can only assume this is because behind the scenes the daemon is using
> GET and hitting the same url limit (but the error is being swallowed).
>
> Significantly increasing the max http header length in jetty resolved the
> issue so this feels like a bit of a bug?
>
>
> On 22/03/2021, 15:37, "Karl Stoney" <Ka...@autotrader.co.uk.INVALID>
> wrote:
>
> So for context we have 900x fields on Collection one and have removed
> some 250 fields from the schema and want to reindex into collection2.
> We're trying to have a process where we can easily remove fields and
> reindex without too much coding overhead. Therefore, we were simply using
> the default `fl=*:*` from the first collection as the assumption was that
> when we try and save the document to collection2, the additional fields
> would just be ignored. This wasn't the case.
>
> Subsequently to work around this now we read the schema.xml for
> collection2 and build up a `fl` to pass to REINDEXCOLLECTION which only
> includes the fields in collection2's schema. Which works.
>
>
> On 22/03/2021, 13:20, "David Hastings" <ha...@gmail.com>
> wrote:
>
> >Surely this field should simply just be ignored?
>
> why would solr ignore this field if you're trying to index to it?
> can't
> you change your indexer to remove these fields as well? solr will
> try to
> do what its told, and if its told to do something bad it will
> simply fail,
> you dont want it to ignore errors or bad indexing
>
> On Mon, Mar 22, 2021 at 9:15 AM David Smiley <ds...@apache.org>
> wrote:
>
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_8%2Fcollection-management.html%23reindexcollection&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513742368%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5c6Do5%2FRPNVFdgdgm7kKw97O3xgvyBiMOhJcXDupP9o%3D&reserved=0
> >
> > See the "fl" param
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> >
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidwsmiley&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ATyR1bkLE8Mt9F3oreaVmToKGC6Hzl6RvolmbJA997w%3D&reserved=0
> >
> >
> > On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
> > <Ka...@autotrader.co.uk.invalid> wrote:
> >
> > > Hi,
> > > Sorry for all the questions recently…
> > >
> > > So as per
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_0%2Freindexing.html&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tJ74flsz5IxGN72KI9A4Z%2BurJHRz1QXSpENBlanh%2Fv4%3D&reserved=0;
> we’re
> > trying
> > > to remove a load of fields. Subsequently we’ve created a new
> collection
> > > with the new schema and we’re attempting to reindex from old
> to new.
> > > There’s about 216 fields in total being removed…
> > >
> > > The REINDEX fails though, because the field has been removed:
> > >
> > > 2:52:31.508
> > >
> >
> [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> > > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1
> r:core_node2] ERROR
> > > org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal
> Error in
> > > DaemonStream: at-uk-003
> > >
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > > Error from server at
> > >
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-0.search-solr.svc.cluster.local%2Fsolr%2Fat-uk-003&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eQIWOrab8PPiOQSXvcOMPMJavUCm9KFVEHaKcbQ7v9M%3D&reserved=0:
> ERROR:
> > > [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> > > 'OEM_ENGINE_MARKETING'
> > >
> > > Surely this field should simply just be ignored?
> > >
> > > I can’t see any way of using REINDEXCOLLECTION to reindex data
> from one
> > > collection to another where we have removed fields.
> > >
> > > Any input would be appreciated.
> > > Unless expressly stated otherwise in this email, this e-mail
> is sent on
> > > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson
> Place,
> > > Manchester, Lancashire, M15 4FN (Registered in England No.
> 03909628).
> > Auto
> > > Trader Limited is part of the Auto Trader Group Plc group.
> This email and
> > > any files transmitted with it are confidential and may be
> legally
> > > privileged, and intended solely for the use of the individual
> or entity
> > to
> > > whom they are addressed. If you have received this email in
> error please
> > > notify the sender. This email message has been swept for the
> presence of
> > > computer viruses.
> > >
> >
>
> Unless expressly stated otherwise in this email, this e-mail is sent
> on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
> Trader Limited is part of the Auto Trader Group Plc group. This email and
> any files transmitted with it are confidential and may be legally
> privileged, and intended solely for the use of the individual or entity to
> whom they are addressed. If you have received this email in error please
> notify the sender. This email message has been swept for the presence of
> computer viruses.
>
> Unless expressly stated otherwise in this email, this e-mail is sent on
> behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
> Trader Limited is part of the Auto Trader Group Plc group. This email and
> any files transmitted with it are confidential and may be legally
> privileged, and intended solely for the use of the individual or entity to
> whom they are addressed. If you have received this email in error please
> notify the sender. This email message has been swept for the presence of
> computer viruses.
>
Re: REINDEXCOLLECTION unknown field
Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.
Interestingly enough the next issue we hit is the `fl=750 fields` == too big, so we switched to using POST/x-url-form-encoded for REINDEXCOLLECTION which accepts the request, but then it silently fails (no logs in solr, just doesn't work).
I can only assume this is because behind the scenes the daemon is using GET and hitting the same url limit (but the error is being swallowed).
Significantly increasing the max http header length in jetty resolved the issue so this feels like a bit of a bug?
On 22/03/2021, 15:37, "Karl Stoney" <Ka...@autotrader.co.uk.INVALID> wrote:
So for context we have 900x fields on Collection one and have removed some 250 fields from the schema and want to reindex into collection2. We're trying to have a process where we can easily remove fields and reindex without too much coding overhead. Therefore, we were simply using the default `fl=*:*` from the first collection as the assumption was that when we try and save the document to collection2, the additional fields would just be ignored. This wasn't the case.
Subsequently to work around this now we read the schema.xml for collection2 and build up a `fl` to pass to REINDEXCOLLECTION which only includes the fields in collection2's schema. Which works.
On 22/03/2021, 13:20, "David Hastings" <ha...@gmail.com> wrote:
>Surely this field should simply just be ignored?
why would solr ignore this field if you're trying to index to it? can't
you change your indexer to remove these fields as well? solr will try to
do what its told, and if its told to do something bad it will simply fail,
you dont want it to ignore errors or bad indexing
On Mon, Mar 22, 2021 at 9:15 AM David Smiley <ds...@apache.org> wrote:
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_8%2Fcollection-management.html%23reindexcollection&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513742368%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5c6Do5%2FRPNVFdgdgm7kKw97O3xgvyBiMOhJcXDupP9o%3D&reserved=0
>
> See the "fl" param
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidwsmiley&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ATyR1bkLE8Mt9F3oreaVmToKGC6Hzl6RvolmbJA997w%3D&reserved=0
>
>
> On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
> <Ka...@autotrader.co.uk.invalid> wrote:
>
> > Hi,
> > Sorry for all the questions recently…
> >
> > So as per https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_0%2Freindexing.html&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tJ74flsz5IxGN72KI9A4Z%2BurJHRz1QXSpENBlanh%2Fv4%3D&reserved=0; we’re
> trying
> > to remove a load of fields. Subsequently we’ve created a new collection
> > with the new schema and we’re attempting to reindex from old to new.
> > There’s about 216 fields in total being removed…
> >
> > The REINDEX fails though, because the field has been removed:
> >
> > 2:52:31.508
> >
> [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR
> > org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in
> > DaemonStream: at-uk-003
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at
> > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-0.search-solr.svc.cluster.local%2Fsolr%2Fat-uk-003&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eQIWOrab8PPiOQSXvcOMPMJavUCm9KFVEHaKcbQ7v9M%3D&reserved=0: ERROR:
> > [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> > 'OEM_ENGINE_MARKETING'
> >
> > Surely this field should simply just be ignored?
> >
> > I can’t see any way of using REINDEXCOLLECTION to reindex data from one
> > collection to another where we have removed fields.
> >
> > Any input would be appreciated.
> > Unless expressly stated otherwise in this email, this e-mail is sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628).
> Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or entity
> to
> > whom they are addressed. If you have received this email in error please
> > notify the sender. This email message has been swept for the presence of
> > computer viruses.
> >
>
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Re: REINDEXCOLLECTION unknown field
Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.
So for context we have 900x fields on Collection one and have removed some 250 fields from the schema and want to reindex into collection2. We're trying to have a process where we can easily remove fields and reindex without too much coding overhead. Therefore, we were simply using the default `fl=*:*` from the first collection as the assumption was that when we try and save the document to collection2, the additional fields would just be ignored. This wasn't the case.
Subsequently to work around this now we read the schema.xml for collection2 and build up a `fl` to pass to REINDEXCOLLECTION which only includes the fields in collection2's schema. Which works.
On 22/03/2021, 13:20, "David Hastings" <ha...@gmail.com> wrote:
>Surely this field should simply just be ignored?
why would solr ignore this field if you're trying to index to it? can't
you change your indexer to remove these fields as well? solr will try to
do what its told, and if its told to do something bad it will simply fail,
you dont want it to ignore errors or bad indexing
On Mon, Mar 22, 2021 at 9:15 AM David Smiley <ds...@apache.org> wrote:
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_8%2Fcollection-management.html%23reindexcollection&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C7edb998b270540f5158e08d8ed3548b2%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520160397668908%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EEEVXmXhmAYOhjh5GlRz%2BOnjsOkKNVEsNylpVexmbEM%3D&reserved=0
>
> See the "fl" param
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidwsmiley&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C7edb998b270540f5158e08d8ed3548b2%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520160397668908%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6Sj5I7Zi9N63ON73luJsiKo%2FKUk63RunO7UDDdQDUw0%3D&reserved=0
>
>
> On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
> <Ka...@autotrader.co.uk.invalid> wrote:
>
> > Hi,
> > Sorry for all the questions recently…
> >
> > So as per https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_0%2Freindexing.html&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C7edb998b270540f5158e08d8ed3548b2%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520160397668908%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1hfrG3lAopYZZ97VCTFLzE3HunM9AulqvBr4yhKR%2FuE%3D&reserved=0; we’re
> trying
> > to remove a load of fields. Subsequently we’ve created a new collection
> > with the new schema and we’re attempting to reindex from old to new.
> > There’s about 216 fields in total being removed…
> >
> > The REINDEX fails though, because the field has been removed:
> >
> > 2:52:31.508
> >
> [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR
> > org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in
> > DaemonStream: at-uk-003
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at
> > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-0.search-solr.svc.cluster.local%2Fsolr%2Fat-uk-003&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C7edb998b270540f5158e08d8ed3548b2%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520160397668908%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FkTABhO21E7lGnBO3y%2BetaCMFNTuyqiuQU%2FaYP%2FmrMU%3D&reserved=0: ERROR:
> > [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> > 'OEM_ENGINE_MARKETING'
> >
> > Surely this field should simply just be ignored?
> >
> > I can’t see any way of using REINDEXCOLLECTION to reindex data from one
> > collection to another where we have removed fields.
> >
> > Any input would be appreciated.
> > Unless expressly stated otherwise in this email, this e-mail is sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628).
> Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or entity
> to
> > whom they are addressed. If you have received this email in error please
> > notify the sender. This email message has been swept for the presence of
> > computer viruses.
> >
>
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Re: REINDEXCOLLECTION unknown field
Posted by David Hastings <ha...@gmail.com>.
>Surely this field should simply just be ignored?
why would solr ignore this field if you're trying to index to it? can't
you change your indexer to remove these fields as well? solr will try to
do what its told, and if its told to do something bad it will simply fail,
you dont want it to ignore errors or bad indexing
On Mon, Mar 22, 2021 at 9:15 AM David Smiley <ds...@apache.org> wrote:
>
> https://solr.apache.org/guide/8_8/collection-management.html#reindexcollection
>
> See the "fl" param
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
> <Ka...@autotrader.co.uk.invalid> wrote:
>
> > Hi,
> > Sorry for all the questions recently…
> >
> > So as per https://solr.apache.org/guide/8_0/reindexing.html; we’re
> trying
> > to remove a load of fields. Subsequently we’ve created a new collection
> > with the new schema and we’re attempting to reindex from old to new.
> > There’s about 216 fields in total being removed…
> >
> > The REINDEX fails though, because the field has been removed:
> >
> > 2:52:31.508
> >
> [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR
> > org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in
> > DaemonStream: at-uk-003
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at
> > http://solr-0.search-solr.svc.cluster.local:80/solr/at-uk-003: ERROR:
> > [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> > 'OEM_ENGINE_MARKETING'
> >
> > Surely this field should simply just be ignored?
> >
> > I can’t see any way of using REINDEXCOLLECTION to reindex data from one
> > collection to another where we have removed fields.
> >
> > Any input would be appreciated.
> > Unless expressly stated otherwise in this email, this e-mail is sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628).
> Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or entity
> to
> > whom they are addressed. If you have received this email in error please
> > notify the sender. This email message has been swept for the presence of
> > computer viruses.
> >
>
Re: REINDEXCOLLECTION unknown field
Posted by David Smiley <ds...@apache.org>.
https://solr.apache.org/guide/8_8/collection-management.html#reindexcollection
See the "fl" param
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
<Ka...@autotrader.co.uk.invalid> wrote:
> Hi,
> Sorry for all the questions recently…
>
> So as per https://solr.apache.org/guide/8_0/reindexing.html; we’re trying
> to remove a load of fields. Subsequently we’ve created a new collection
> with the new schema and we’re attempting to reindex from old to new.
> There’s about 216 fields in total being removed…
>
> The REINDEX fails though, because the field has been removed:
>
> 2:52:31.508
> [DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2] ERROR
> org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in
> DaemonStream: at-uk-003
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at
> http://solr-0.search-solr.svc.cluster.local:80/solr/at-uk-003: ERROR:
> [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> 'OEM_ENGINE_MARKETING'
>
> Surely this field should simply just be ignored?
>
> I can’t see any way of using REINDEXCOLLECTION to reindex data from one
> collection to another where we have removed fields.
>
> Any input would be appreciated.
> Unless expressly stated otherwise in this email, this e-mail is sent on
> behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
> Trader Limited is part of the Auto Trader Group Plc group. This email and
> any files transmitted with it are confidential and may be legally
> privileged, and intended solely for the use of the individual or entity to
> whom they are addressed. If you have received this email in error please
> notify the sender. This email message has been swept for the presence of
> computer viruses.
>