You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@karaf.apache.org by James Ziesig <ja...@broadcom.com> on 2024/03/28 03:48:43 UTC
Race between fileinstall and configadmin?
Hello,
I have been tracking a problem in recent versions of Karaf where random
configuration files in the etc directory are corrupted (or re-initialized)
during karaf initialization. Most of the times that the problem occurs all
of the property settings are removed from the file, but sometimes the file
is re-initialized with properties in it (just not the values that were
present before startup). This only seems to happen when the
apache-karaf-4.x.x/data directory is cleared out prior to startup
(something we do on install/upgrade or after an ungraceful shutdown).
The issue seems to occur 3-5% of the time that Karaf is re-initialized.
I have seen https://issues.apache.org/jira/browse/KARAF-6866 which covers
an example of the problem, but doesn't cover all cases. It led me to try a
"fix" where I alter startup.properties to start fileinstall at level 9 so
it starts before configadmin. I have had 100% success with well over 300
restarts with this configuration, but I don't know if that rearrangement
may lead to some other issue. Is this a valid solution?
This problem has occurred on various versions of CentOS/RHEL/Rocky Linux
(primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green
Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
I have attached a simple bash script for reproduction.
Reproduction steps:
Extract apache-karaf-4.4.3.tar.gz.
cd apache-karaf-4.4.3
cp -r etc etc_orig
rm -Rf data/*
bin/start
diff -r -w etc etc_orig/
check if anything was already overwritten and reset etc with etc_orig if it
was and then repeat until etc looks good.
cp -r etc etc_backup (for use by script)
copy loop.sh to apache-karaf-4.4.3
execute loop.sh
Please let me know if you need any additional info.
Thanks in advance,
Jim Ziesig
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
Re: Race between fileinstall and configadmin?
Posted by James Ziesig <ja...@broadcom.com>.
We are seeing the problem primarily on GCP n2d-standard-4 systems. Some
are using a custom config with only 8GB RAM, but my testing was done on a
stock n2d-standard-4 VM with a standard 100GB disk.
Thanks,
Jim
On Sun, Mar 31, 2024 at 1:11 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Thanks for the details Jim.
>
> Does it happen in a docker container or directly on a machine ? How
> performant is the machine ? (in order for me to reproduce).
>
> Thanks !
> Regards
> JB
>
> On Thu, Mar 28, 2024 at 1:44 PM James Ziesig <ja...@broadcom.com>
> wrote:
> >
> > Hi JB,
> >
> > Thanks for looking into this. Yes, I know that some of the files are
> immune to the problem. I could see that only some of the files in the etc
> directory were modified between startup and the timestamp of the first log
> with ms precision:
> >
> > "INFO | s4j.pax.logging) | 2024-03-27T20:25:52,051 |
> EventAdminConfigurationNotifier | .EventAdminConfigurationNotifier 48 |
> gging.pax-logging-log4j2 | | Sending Event Admin notification
> (configuration successful) to org/ops4j/pax/logging/Configuration"
> >
> > I couldn't figure out how to enable fileinstall debug logging to see if
> it was the one modifying the files, though.
> >
> > I've seen the problem happen with jmx, command, logging, feature
> service, and our own configuration files.
> >
> > Hopefully you can identify the problem.
> >
> > Thanks again!
> >
> > Jim Ziesig
> >
> > On Thu, Mar 28, 2024 at 1:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >>
> >> Hi Jim
> >>
> >> We did some improvements of potential race conditions between features
> >> and config admin services. Remember some files in etc are not managed
> >> by config admin (for instance startup.properties is used before the
> >> services are actually started and not managed by config admin).
> >>
> >> Let me try to reproduce your test case and I will let you know.
> >>
> >> Thanks,
> >> Regards
> >> JB
> >>
> >> On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com>
> wrote:
> >> >
> >> > Hello,
> >> >
> >> > I have been tracking a problem in recent versions of Karaf where
> random configuration files in the etc directory are corrupted (or
> re-initialized) during karaf initialization. Most of the times that the
> problem occurs all of the property settings are removed from the file, but
> sometimes the file is re-initialized with properties in it (just not the
> values that were present before startup). This only seems to happen when
> the apache-karaf-4.x.x/data directory is cleared out prior to startup
> (something we do on install/upgrade or after an ungraceful shutdown). The
> issue seems to occur 3-5% of the time that Karaf is re-initialized.
> >> >
> >> > I have seen https://issues.apache.org/jira/browse/KARAF-6866 which
> covers an example of the problem, but doesn't cover all cases. It led me
> to try a "fix" where I alter startup.properties to start fileinstall at
> level 9 so it starts before configadmin. I have had 100% success with well
> over 300 restarts with this configuration, but I don't know if that
> rearrangement may lead to some other issue. Is this a valid solution?
> >> >
> >> > This problem has occurred on various versions of CentOS/RHEL/Rocky
> Linux (primarily 8.x). My testing was done on Rocky Linux release 8.9
> (Green Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
> >> >
> >> > I have attached a simple bash script for reproduction.
> >> >
> >> > Reproduction steps:
> >> > Extract apache-karaf-4.4.3.tar.gz.
> >> > cd apache-karaf-4.4.3
> >> > cp -r etc etc_orig
> >> > rm -Rf data/*
> >> > bin/start
> >> > diff -r -w etc etc_orig/
> >> > check if anything was already overwritten and reset etc with etc_orig
> if it was and then repeat until etc looks good.
> >> > cp -r etc etc_backup (for use by script)
> >> > copy loop.sh to apache-karaf-4.4.3
> >> > execute loop.sh
> >> >
> >> > Please let me know if you need any additional info.
> >> >
> >> > Thanks in advance,
> >> >
> >> > Jim Ziesig
> >> >
> >> > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
> >
> >
> > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
Re: Race between fileinstall and configadmin?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks for the details Jim.
Does it happen in a docker container or directly on a machine ? How
performant is the machine ? (in order for me to reproduce).
Thanks !
Regards
JB
On Thu, Mar 28, 2024 at 1:44 PM James Ziesig <ja...@broadcom.com> wrote:
>
> Hi JB,
>
> Thanks for looking into this. Yes, I know that some of the files are immune to the problem. I could see that only some of the files in the etc directory were modified between startup and the timestamp of the first log with ms precision:
>
> "INFO | s4j.pax.logging) | 2024-03-27T20:25:52,051 | EventAdminConfigurationNotifier | .EventAdminConfigurationNotifier 48 | gging.pax-logging-log4j2 | | Sending Event Admin notification (configuration successful) to org/ops4j/pax/logging/Configuration"
>
> I couldn't figure out how to enable fileinstall debug logging to see if it was the one modifying the files, though.
>
> I've seen the problem happen with jmx, command, logging, feature service, and our own configuration files.
>
> Hopefully you can identify the problem.
>
> Thanks again!
>
> Jim Ziesig
>
> On Thu, Mar 28, 2024 at 1:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Hi Jim
>>
>> We did some improvements of potential race conditions between features
>> and config admin services. Remember some files in etc are not managed
>> by config admin (for instance startup.properties is used before the
>> services are actually started and not managed by config admin).
>>
>> Let me try to reproduce your test case and I will let you know.
>>
>> Thanks,
>> Regards
>> JB
>>
>> On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com> wrote:
>> >
>> > Hello,
>> >
>> > I have been tracking a problem in recent versions of Karaf where random configuration files in the etc directory are corrupted (or re-initialized) during karaf initialization. Most of the times that the problem occurs all of the property settings are removed from the file, but sometimes the file is re-initialized with properties in it (just not the values that were present before startup). This only seems to happen when the apache-karaf-4.x.x/data directory is cleared out prior to startup (something we do on install/upgrade or after an ungraceful shutdown). The issue seems to occur 3-5% of the time that Karaf is re-initialized.
>> >
>> > I have seen https://issues.apache.org/jira/browse/KARAF-6866 which covers an example of the problem, but doesn't cover all cases. It led me to try a "fix" where I alter startup.properties to start fileinstall at level 9 so it starts before configadmin. I have had 100% success with well over 300 restarts with this configuration, but I don't know if that rearrangement may lead to some other issue. Is this a valid solution?
>> >
>> > This problem has occurred on various versions of CentOS/RHEL/Rocky Linux (primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
>> >
>> > I have attached a simple bash script for reproduction.
>> >
>> > Reproduction steps:
>> > Extract apache-karaf-4.4.3.tar.gz.
>> > cd apache-karaf-4.4.3
>> > cp -r etc etc_orig
>> > rm -Rf data/*
>> > bin/start
>> > diff -r -w etc etc_orig/
>> > check if anything was already overwritten and reset etc with etc_orig if it was and then repeat until etc looks good.
>> > cp -r etc etc_backup (for use by script)
>> > copy loop.sh to apache-karaf-4.4.3
>> > execute loop.sh
>> >
>> > Please let me know if you need any additional info.
>> >
>> > Thanks in advance,
>> >
>> > Jim Ziesig
>> >
>> > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
>
>
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
Re: Race between fileinstall and configadmin?
Posted by James Ziesig <ja...@broadcom.com>.
Hi JB,
Thanks for looking into this. Yes, I know that some of the files are
immune to the problem. I could see that only some of the files in the etc
directory were modified between startup and the timestamp of the first log
with ms precision:
"INFO | s4j.pax.logging) | 2024-03-27T20:25:52,051 |
EventAdminConfigurationNotifier | .EventAdminConfigurationNotifier 48 |
gging.pax-logging-log4j2 | | Sending Event Admin notification
(configuration successful) to org/ops4j/pax/logging/Configuration"
I couldn't figure out how to enable fileinstall debug logging to see if it
was the one modifying the files, though.
I've seen the problem happen with jmx, command, logging, feature service,
and our own configuration files.
Hopefully you can identify the problem.
Thanks again!
Jim Ziesig
On Thu, Mar 28, 2024 at 1:23 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Jim
>
> We did some improvements of potential race conditions between features
> and config admin services. Remember some files in etc are not managed
> by config admin (for instance startup.properties is used before the
> services are actually started and not managed by config admin).
>
> Let me try to reproduce your test case and I will let you know.
>
> Thanks,
> Regards
> JB
>
> On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com>
> wrote:
> >
> > Hello,
> >
> > I have been tracking a problem in recent versions of Karaf where random
> configuration files in the etc directory are corrupted (or re-initialized)
> during karaf initialization. Most of the times that the problem occurs all
> of the property settings are removed from the file, but sometimes the file
> is re-initialized with properties in it (just not the values that were
> present before startup). This only seems to happen when the
> apache-karaf-4.x.x/data directory is cleared out prior to startup
> (something we do on install/upgrade or after an ungraceful shutdown). The
> issue seems to occur 3-5% of the time that Karaf is re-initialized.
> >
> > I have seen https://issues.apache.org/jira/browse/KARAF-6866 which
> covers an example of the problem, but doesn't cover all cases. It led me
> to try a "fix" where I alter startup.properties to start fileinstall at
> level 9 so it starts before configadmin. I have had 100% success with well
> over 300 restarts with this configuration, but I don't know if that
> rearrangement may lead to some other issue. Is this a valid solution?
> >
> > This problem has occurred on various versions of CentOS/RHEL/Rocky Linux
> (primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green
> Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
> >
> > I have attached a simple bash script for reproduction.
> >
> > Reproduction steps:
> > Extract apache-karaf-4.4.3.tar.gz.
> > cd apache-karaf-4.4.3
> > cp -r etc etc_orig
> > rm -Rf data/*
> > bin/start
> > diff -r -w etc etc_orig/
> > check if anything was already overwritten and reset etc with etc_orig if
> it was and then repeat until etc looks good.
> > cp -r etc etc_backup (for use by script)
> > copy loop.sh to apache-karaf-4.4.3
> > execute loop.sh
> >
> > Please let me know if you need any additional info.
> >
> > Thanks in advance,
> >
> > Jim Ziesig
> >
> > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
Re: Race between fileinstall and configadmin?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
KARAF-7389 should be fixed on Karaf 4.4.1, but Jim seems to have the
issue with 4.4.3.
So let me try to reproduce first.
Regards
JB
On Thu, Mar 28, 2024 at 8:17 AM Grzegorz Grzybek <gr...@gmail.com> wrote:
>
> Hello
>
> I can't check right now (Easter holidays) , but I can point you to similar (same?) issue: https://issues.apache.org/jira/browse/KARAF-7389
>
> regards
> Grzegorz Grzybek
>
> czw., 28 mar 2024 o 06:23 Jean-Baptiste Onofré <jb...@nanthrax.net> napisał(a):
>>
>> Hi Jim
>>
>> We did some improvements of potential race conditions between features
>> and config admin services. Remember some files in etc are not managed
>> by config admin (for instance startup.properties is used before the
>> services are actually started and not managed by config admin).
>>
>> Let me try to reproduce your test case and I will let you know.
>>
>> Thanks,
>> Regards
>> JB
>>
>> On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com> wrote:
>> >
>> > Hello,
>> >
>> > I have been tracking a problem in recent versions of Karaf where random configuration files in the etc directory are corrupted (or re-initialized) during karaf initialization. Most of the times that the problem occurs all of the property settings are removed from the file, but sometimes the file is re-initialized with properties in it (just not the values that were present before startup). This only seems to happen when the apache-karaf-4.x.x/data directory is cleared out prior to startup (something we do on install/upgrade or after an ungraceful shutdown). The issue seems to occur 3-5% of the time that Karaf is re-initialized.
>> >
>> > I have seen https://issues.apache.org/jira/browse/KARAF-6866 which covers an example of the problem, but doesn't cover all cases. It led me to try a "fix" where I alter startup.properties to start fileinstall at level 9 so it starts before configadmin. I have had 100% success with well over 300 restarts with this configuration, but I don't know if that rearrangement may lead to some other issue. Is this a valid solution?
>> >
>> > This problem has occurred on various versions of CentOS/RHEL/Rocky Linux (primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
>> >
>> > I have attached a simple bash script for reproduction.
>> >
>> > Reproduction steps:
>> > Extract apache-karaf-4.4.3.tar.gz.
>> > cd apache-karaf-4.4.3
>> > cp -r etc etc_orig
>> > rm -Rf data/*
>> > bin/start
>> > diff -r -w etc etc_orig/
>> > check if anything was already overwritten and reset etc with etc_orig if it was and then repeat until etc looks good.
>> > cp -r etc etc_backup (for use by script)
>> > copy loop.sh to apache-karaf-4.4.3
>> > execute loop.sh
>> >
>> > Please let me know if you need any additional info.
>> >
>> > Thanks in advance,
>> >
>> > Jim Ziesig
>> >
>> > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
Re: Race between fileinstall and configadmin?
Posted by Grzegorz Grzybek <gr...@gmail.com>.
Hello
I can't check right now (Easter holidays) , but I can point you to similar
(same?) issue: https://issues.apache.org/jira/browse/KARAF-7389
regards
Grzegorz Grzybek
czw., 28 mar 2024 o 06:23 Jean-Baptiste Onofré <jb...@nanthrax.net> napisał(a):
> Hi Jim
>
> We did some improvements of potential race conditions between features
> and config admin services. Remember some files in etc are not managed
> by config admin (for instance startup.properties is used before the
> services are actually started and not managed by config admin).
>
> Let me try to reproduce your test case and I will let you know.
>
> Thanks,
> Regards
> JB
>
> On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com>
> wrote:
> >
> > Hello,
> >
> > I have been tracking a problem in recent versions of Karaf where random
> configuration files in the etc directory are corrupted (or re-initialized)
> during karaf initialization. Most of the times that the problem occurs all
> of the property settings are removed from the file, but sometimes the file
> is re-initialized with properties in it (just not the values that were
> present before startup). This only seems to happen when the
> apache-karaf-4.x.x/data directory is cleared out prior to startup
> (something we do on install/upgrade or after an ungraceful shutdown). The
> issue seems to occur 3-5% of the time that Karaf is re-initialized.
> >
> > I have seen https://issues.apache.org/jira/browse/KARAF-6866 which
> covers an example of the problem, but doesn't cover all cases. It led me
> to try a "fix" where I alter startup.properties to start fileinstall at
> level 9 so it starts before configadmin. I have had 100% success with well
> over 300 restarts with this configuration, but I don't know if that
> rearrangement may lead to some other issue. Is this a valid solution?
> >
> > This problem has occurred on various versions of CentOS/RHEL/Rocky Linux
> (primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green
> Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
> >
> > I have attached a simple bash script for reproduction.
> >
> > Reproduction steps:
> > Extract apache-karaf-4.4.3.tar.gz.
> > cd apache-karaf-4.4.3
> > cp -r etc etc_orig
> > rm -Rf data/*
> > bin/start
> > diff -r -w etc etc_orig/
> > check if anything was already overwritten and reset etc with etc_orig if
> it was and then repeat until etc looks good.
> > cp -r etc etc_backup (for use by script)
> > copy loop.sh to apache-karaf-4.4.3
> > execute loop.sh
> >
> > Please let me know if you need any additional info.
> >
> > Thanks in advance,
> >
> > Jim Ziesig
> >
> > This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
Re: Race between fileinstall and configadmin?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Jim
We did some improvements of potential race conditions between features
and config admin services. Remember some files in etc are not managed
by config admin (for instance startup.properties is used before the
services are actually started and not managed by config admin).
Let me try to reproduce your test case and I will let you know.
Thanks,
Regards
JB
On Thu, Mar 28, 2024 at 4:48 AM James Ziesig <ja...@broadcom.com> wrote:
>
> Hello,
>
> I have been tracking a problem in recent versions of Karaf where random configuration files in the etc directory are corrupted (or re-initialized) during karaf initialization. Most of the times that the problem occurs all of the property settings are removed from the file, but sometimes the file is re-initialized with properties in it (just not the values that were present before startup). This only seems to happen when the apache-karaf-4.x.x/data directory is cleared out prior to startup (something we do on install/upgrade or after an ungraceful shutdown). The issue seems to occur 3-5% of the time that Karaf is re-initialized.
>
> I have seen https://issues.apache.org/jira/browse/KARAF-6866 which covers an example of the problem, but doesn't cover all cases. It led me to try a "fix" where I alter startup.properties to start fileinstall at level 9 so it starts before configadmin. I have had 100% success with well over 300 restarts with this configuration, but I don't know if that rearrangement may lead to some other issue. Is this a valid solution?
>
> This problem has occurred on various versions of CentOS/RHEL/Rocky Linux (primarily 8.x). My testing was done on Rocky Linux release 8.9 (Green Obsidian), using OpenJDK 64-Bit Server VM Temurin-17.0.10+7.
>
> I have attached a simple bash script for reproduction.
>
> Reproduction steps:
> Extract apache-karaf-4.4.3.tar.gz.
> cd apache-karaf-4.4.3
> cp -r etc etc_orig
> rm -Rf data/*
> bin/start
> diff -r -w etc etc_orig/
> check if anything was already overwritten and reset etc with etc_orig if it was and then repeat until etc looks good.
> cp -r etc etc_backup (for use by script)
> copy loop.sh to apache-karaf-4.4.3
> execute loop.sh
>
> Please let me know if you need any additional info.
>
> Thanks in advance,
>
> Jim Ziesig
>
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.