You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Uwe Schindler <uw...@thetaphi.de> on 2025/03/18 16:52:57 UTC
Policeman Jenkins => new hardware
Moin moin,
Policeman Jenkins got new hardware yesterday - no functional changes.
Background: The old server had some strange problems with the networking
adaptor (Intel's "igb" kernel driver) about "Detected Tx Unit Hang".
This caused some short downtimes and the monitoring complained all the
time about lost pings which drove me crazy at weekend. It worked better
after a restart and also with downgrade of kernel, but as I was about to
replace the machine by a newer one, I ordered a replacement to new
Hardware version (previously it was Hetzner AX51-NVME; now it is:
Hetzner AX52).
The migration was done starting yesterday lunch time europe (12:00 CET)
in the by booting the new server in the datacenter's recovery
environment booted from network on both servers with a temporary IP and
then mounting both root disks and doing a large rsync (with checksums,
external attributes, numeric uid/gid and delete option). Luckily this
worked with the old server (the Intel Adapter did not break). The whole
downtime should have taken only 1 to 1.5 hours (the time copy with 1
GBits and reconfig needs), but unfortunately the PCIexpress on the new
server complained about (recoverable) errors on the NVME communications.
After some support roundtrips (they first replaced only the failing NVME
controller which did not help), the replaced the whole server.
At 18:30 CET, I started copy to new server again and all went well,
dmesg showed no PCI express checksum errors. Finally, after fixing boot
(the old server used MBR the new one EFI), the server was mounted at the
original location by the team and all IPv4 adresses and IPv6 network
were available. Since then (approx 20:30 CET), Policeman Jenkins is back
and running.
The TODOs for the future:
* Replace the MacOS VM and update it to a new version (it's
complicated, as it is a "Hackintosh", so it shouldn't be there
according to Apple)
* Possibly migrate away from VirtualBOX to KVM, but it's unclear if
Hackintoshs work there.
Have fun with the new hardware, the builds on Lucene main branch are now
1.5 times faster (10 instead of 15 minutes).
The new hardware is described here:
https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
let's see what comes out. No test failures yet.
vendor_id : AuthenticAMD
cpu family : 25
model : 97
model name : AMD Ryzen 7 7700 8-Core Processor
stepping : 2
microcode : 0xa601209
cpu MHz : 5114.082
cache size : 1024 KB
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
apicid : 15
initial apicid : 15
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl
xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb
bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2
ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms
invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt
clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves
cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16
clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock
nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke
avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg
avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
amd_lbr_pmc_freeze
bugs : sysret_ss_attrs spectre_v1 spectre_v2
spec_store_bypass srso
bogomips : 7585.28
TLB size : 3584 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
# lspci | fgrep -i volati
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe
SSD Controller PM9A1/PM9A3/980PRO
02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe
SSD [Hendrix]
I have no idea why the replacement server has two different NVME SSDs,
but you never know before what you get! From smart info I know that both
SSDs were fresh (6 hours total uptime only).
Uwe
--
Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
eMail: uwe@thetaphi.de
Re: Policeman Jenkins => new hardware
Posted by David Smiley <ds...@apache.org>.
The 10x write factor is probably logs. Solr writes a lot of logs. Most
tests need very little data to write & read.
On Wed, Mar 19, 2025 at 8:16 AM Michael Sokolov <ms...@gmail.com> wrote:
> Woohoo! thanks Uwe; exciting you were able to get 2x the lifespan of
> the drives. Let's go for 4x this time!
>
> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > Moin moin,
> >
> > Policeman Jenkins got new hardware yesterday - no functional changes.
> >
> > Background: The old server had some strange problems with the networking
> adaptor (Intel's "igb" kernel driver) about "Detected Tx Unit Hang". This
> caused some short downtimes and the monitoring complained all the time
> about lost pings which drove me crazy at weekend. It worked better after a
> restart and also with downgrade of kernel, but as I was about to replace
> the machine by a newer one, I ordered a replacement to new Hardware version
> (previously it was Hetzner AX51-NVME; now it is: Hetzner AX52).
> >
> > The migration was done starting yesterday lunch time europe (12:00 CET)
> in the by booting the new server in the datacenter's recovery environment
> booted from network on both servers with a temporary IP and then mounting
> both root disks and doing a large rsync (with checksums, external
> attributes, numeric uid/gid and delete option). Luckily this worked with
> the old server (the Intel Adapter did not break). The whole downtime should
> have taken only 1 to 1.5 hours (the time copy with 1 GBits and reconfig
> needs), but unfortunately the PCIexpress on the new server complained about
> (recoverable) errors on the NVME communications. After some support
> roundtrips (they first replaced only the failing NVME controller which did
> not help), the replaced the whole server.
> >
> > At 18:30 CET, I started copy to new server again and all went well,
> dmesg showed no PCI express checksum errors. Finally, after fixing boot
> (the old server used MBR the new one EFI), the server was mounted at the
> original location by the team and all IPv4 adresses and IPv6 network were
> available. Since then (approx 20:30 CET), Policeman Jenkins is back and
> running.
> >
> > The TODOs for the future:
> >
> > Replace the MacOS VM and update it to a new version (it's complicated,
> as it is a "Hackintosh", so it shouldn't be there according to Apple)
> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
> >
> > Have fun with the new hardware, the builds on Lucene main branch are now
> 1.5 times faster (10 instead of 15 minutes).
> >
> > The new hardware is described here:
> https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
> let's see what comes out. No test failures yet.
> >
> > vendor_id : AuthenticAMD
> > cpu family : 25
> > model : 97
> > model name : AMD Ryzen 7 7700 8-Core Processor
> > stepping : 2
> > microcode : 0xa601209
> > cpu MHz : 5114.082
> > cache size : 1024 KB
> > physical id : 0
> > siblings : 16
> > core id : 7
> > cpu cores : 8
> > apicid : 15
> > initial apicid : 15
> > fpu : yes
> > fpu_exception : yes
> > cpuid level : 16
> > wp : yes
> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
> pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology
> nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3
> fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm
> cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw
> ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx
> cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp
> ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a
> avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni
> avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
> cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr
> rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
> flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
> v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
> vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov
> succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> > bugs : sysret_ss_attrs spectre_v1 spectre_v2
> spec_store_bypass srso
> > bogomips : 7585.28
> > TLB size : 3584 4K pages
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 48 bits physical, 48 bits virtual
> > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> >
> > # lspci | fgrep -i volati
> > 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe
> SSD Controller PM9A1/PM9A3/980PRO
> > 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe
> SSD [Hendrix]
> >
> > I have no idea why the replacement server has two different NVME SSDs,
> but you never know before what you get! From smart info I know that both
> SSDs were fresh (6 hours total uptime only).
> >
> > Uwe
> >
> > --
> >
> > Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Policeman Jenkins => new hardware
Posted by David Smiley <ds...@apache.org>.
The 10x write factor is probably logs. Solr writes a lot of logs. Most
tests need very little data to write & read.
On Wed, Mar 19, 2025 at 8:16 AM Michael Sokolov <ms...@gmail.com> wrote:
> Woohoo! thanks Uwe; exciting you were able to get 2x the lifespan of
> the drives. Let's go for 4x this time!
>
> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > Moin moin,
> >
> > Policeman Jenkins got new hardware yesterday - no functional changes.
> >
> > Background: The old server had some strange problems with the networking
> adaptor (Intel's "igb" kernel driver) about "Detected Tx Unit Hang". This
> caused some short downtimes and the monitoring complained all the time
> about lost pings which drove me crazy at weekend. It worked better after a
> restart and also with downgrade of kernel, but as I was about to replace
> the machine by a newer one, I ordered a replacement to new Hardware version
> (previously it was Hetzner AX51-NVME; now it is: Hetzner AX52).
> >
> > The migration was done starting yesterday lunch time europe (12:00 CET)
> in the by booting the new server in the datacenter's recovery environment
> booted from network on both servers with a temporary IP and then mounting
> both root disks and doing a large rsync (with checksums, external
> attributes, numeric uid/gid and delete option). Luckily this worked with
> the old server (the Intel Adapter did not break). The whole downtime should
> have taken only 1 to 1.5 hours (the time copy with 1 GBits and reconfig
> needs), but unfortunately the PCIexpress on the new server complained about
> (recoverable) errors on the NVME communications. After some support
> roundtrips (they first replaced only the failing NVME controller which did
> not help), the replaced the whole server.
> >
> > At 18:30 CET, I started copy to new server again and all went well,
> dmesg showed no PCI express checksum errors. Finally, after fixing boot
> (the old server used MBR the new one EFI), the server was mounted at the
> original location by the team and all IPv4 adresses and IPv6 network were
> available. Since then (approx 20:30 CET), Policeman Jenkins is back and
> running.
> >
> > The TODOs for the future:
> >
> > Replace the MacOS VM and update it to a new version (it's complicated,
> as it is a "Hackintosh", so it shouldn't be there according to Apple)
> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
> >
> > Have fun with the new hardware, the builds on Lucene main branch are now
> 1.5 times faster (10 instead of 15 minutes).
> >
> > The new hardware is described here:
> https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
> let's see what comes out. No test failures yet.
> >
> > vendor_id : AuthenticAMD
> > cpu family : 25
> > model : 97
> > model name : AMD Ryzen 7 7700 8-Core Processor
> > stepping : 2
> > microcode : 0xa601209
> > cpu MHz : 5114.082
> > cache size : 1024 KB
> > physical id : 0
> > siblings : 16
> > core id : 7
> > cpu cores : 8
> > apicid : 15
> > initial apicid : 15
> > fpu : yes
> > fpu_exception : yes
> > cpuid level : 16
> > wp : yes
> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
> pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology
> nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3
> fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm
> cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw
> ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx
> cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp
> ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a
> avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni
> avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
> cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr
> rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
> flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
> v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
> vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov
> succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> > bugs : sysret_ss_attrs spectre_v1 spectre_v2
> spec_store_bypass srso
> > bogomips : 7585.28
> > TLB size : 3584 4K pages
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 48 bits physical, 48 bits virtual
> > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> >
> > # lspci | fgrep -i volati
> > 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe
> SSD Controller PM9A1/PM9A3/980PRO
> > 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe
> SSD [Hendrix]
> >
> > I have no idea why the replacement server has two different NVME SSDs,
> but you never know before what you get! From smart info I know that both
> SSDs were fresh (6 hours total uptime only).
> >
> > Uwe
> >
> > --
> >
> > Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Policeman Jenkins => new hardware
Posted by Michael Sokolov <ms...@gmail.com>.
Woohoo! thanks Uwe; exciting you were able to get 2x the lifespan of
the drives. Let's go for 4x this time!
On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> Moin moin,
>
> Policeman Jenkins got new hardware yesterday - no functional changes.
>
> Background: The old server had some strange problems with the networking adaptor (Intel's "igb" kernel driver) about "Detected Tx Unit Hang". This caused some short downtimes and the monitoring complained all the time about lost pings which drove me crazy at weekend. It worked better after a restart and also with downgrade of kernel, but as I was about to replace the machine by a newer one, I ordered a replacement to new Hardware version (previously it was Hetzner AX51-NVME; now it is: Hetzner AX52).
>
> The migration was done starting yesterday lunch time europe (12:00 CET) in the by booting the new server in the datacenter's recovery environment booted from network on both servers with a temporary IP and then mounting both root disks and doing a large rsync (with checksums, external attributes, numeric uid/gid and delete option). Luckily this worked with the old server (the Intel Adapter did not break). The whole downtime should have taken only 1 to 1.5 hours (the time copy with 1 GBits and reconfig needs), but unfortunately the PCIexpress on the new server complained about (recoverable) errors on the NVME communications. After some support roundtrips (they first replaced only the failing NVME controller which did not help), the replaced the whole server.
>
> At 18:30 CET, I started copy to new server again and all went well, dmesg showed no PCI express checksum errors. Finally, after fixing boot (the old server used MBR the new one EFI), the server was mounted at the original location by the team and all IPv4 adresses and IPv6 network were available. Since then (approx 20:30 CET), Policeman Jenkins is back and running.
>
> The TODOs for the future:
>
> Replace the MacOS VM and update it to a new version (it's complicated, as it is a "Hackintosh", so it shouldn't be there according to Apple)
> Possibly migrate away from VirtualBOX to KVM, but it's unclear if Hackintoshs work there.
>
> Have fun with the new hardware, the builds on Lucene main branch are now 1.5 times faster (10 instead of 15 minutes).
>
> The new hardware is described here: https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512.... let's see what comes out. No test failures yet.
>
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 97
> model name : AMD Ryzen 7 7700 8-Core Processor
> stepping : 2
> microcode : 0xa601209
> cpu MHz : 5114.082
> cache size : 1024 KB
> physical id : 0
> siblings : 16
> core id : 7
> cpu cores : 8
> apicid : 15
> initial apicid : 15
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
> bogomips : 7585.28
> TLB size : 3584 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> # lspci | fgrep -i volati
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
> 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
>
> I have no idea why the replacement server has two different NVME SSDs, but you never know before what you get! From smart info I know that both SSDs were fresh (6 hours total uptime only).
>
> Uwe
>
> --
>
> Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Policeman Jenkins => new hardware
Posted by Michael Sokolov <ms...@gmail.com>.
Woohoo! thanks Uwe; exciting you were able to get 2x the lifespan of
the drives. Let's go for 4x this time!
On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> Moin moin,
>
> Policeman Jenkins got new hardware yesterday - no functional changes.
>
> Background: The old server had some strange problems with the networking adaptor (Intel's "igb" kernel driver) about "Detected Tx Unit Hang". This caused some short downtimes and the monitoring complained all the time about lost pings which drove me crazy at weekend. It worked better after a restart and also with downgrade of kernel, but as I was about to replace the machine by a newer one, I ordered a replacement to new Hardware version (previously it was Hetzner AX51-NVME; now it is: Hetzner AX52).
>
> The migration was done starting yesterday lunch time europe (12:00 CET) in the by booting the new server in the datacenter's recovery environment booted from network on both servers with a temporary IP and then mounting both root disks and doing a large rsync (with checksums, external attributes, numeric uid/gid and delete option). Luckily this worked with the old server (the Intel Adapter did not break). The whole downtime should have taken only 1 to 1.5 hours (the time copy with 1 GBits and reconfig needs), but unfortunately the PCIexpress on the new server complained about (recoverable) errors on the NVME communications. After some support roundtrips (they first replaced only the failing NVME controller which did not help), the replaced the whole server.
>
> At 18:30 CET, I started copy to new server again and all went well, dmesg showed no PCI express checksum errors. Finally, after fixing boot (the old server used MBR the new one EFI), the server was mounted at the original location by the team and all IPv4 adresses and IPv6 network were available. Since then (approx 20:30 CET), Policeman Jenkins is back and running.
>
> The TODOs for the future:
>
> Replace the MacOS VM and update it to a new version (it's complicated, as it is a "Hackintosh", so it shouldn't be there according to Apple)
> Possibly migrate away from VirtualBOX to KVM, but it's unclear if Hackintoshs work there.
>
> Have fun with the new hardware, the builds on Lucene main branch are now 1.5 times faster (10 instead of 15 minutes).
>
> The new hardware is described here: https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512.... let's see what comes out. No test failures yet.
>
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 97
> model name : AMD Ryzen 7 7700 8-Core Processor
> stepping : 2
> microcode : 0xa601209
> cpu MHz : 5114.082
> cache size : 1024 KB
> physical id : 0
> siblings : 16
> core id : 7
> cpu cores : 8
> apicid : 15
> initial apicid : 15
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
> bogomips : 7585.28
> TLB size : 3584 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> # lspci | fgrep -i volati
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
> 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
>
> I have no idea why the replacement server has two different NVME SSDs, but you never know before what you get! From smart info I know that both SSDs were fresh (6 hours total uptime only).
>
> Uwe
>
> --
>
> Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Policeman Jenkins => new hardware
Posted by Houston Putman <ho...@apache.org>.
That's awesome!
The tests are running really well now. Thanks for all this (continued)
work, it's really appreciated!!
- Houston
On Mon, Mar 24, 2025 at 5:29 AM Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi a small update,
>
> the new virtualization works phantastic. Windows builds are a bit slower
> than others, but this is caused by NTFS and Windows file locking. Basically
> it is not different than with your local WIndows laptop. Macos builds are
> nearly as fast as Linux builds (keep in mind that the number of assigned
> CPUs differs between host and guest VM). This also applies to Solr.
>
> All JDKs were updated on Saturday. So Lucene main builds work.
>
> Everything is using virtio drivers (network and disk images).
> Interestingly, macOs BY DEFAULT ships with drivers for "virtio" networking
> and disks (according to observations by others they do this since Mojave)!
> So the whole thing is as fast as "plain hardware" and real SSD disks (no
> SATA emulation needed):
>
> @Paul Irwin: Because of macos shipping with "virtio drivers", they support
> virtualization officially, also with KVM! They either test their own builds
> in KVM for easier debugging the kernel or they officially support it. The
> "osk" key is not a secret anymore. You could only argue that "I" do not run
> it on native Apple hardware.
>
> Uwe
> Am 22.03.2025 um 00:14 schrieb Uwe Schindler:
>
> Hi,
>
> I converted "Windows VM" (that was easy) and reinstalled a brand-new
> "macOS 15 Sequoia VM" in Policeman Jenkins without using VirtualBOX
> anymore. All is running now as libvirt-based KVM instances. The builds are
> now
> faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaast!!!!!1!!1111!!!!!!!
> :-)
>
> The Windows and macOS VMs now build Lucene and especially Solr in similar
> times. I also extended disk space to 128 GiB per VM.
>
> I also have not seen any hangs of macOS, so the native support by KVM with
> some CPUID adaptions and good wrk by qemu is working well. Thanks to
> https://github.com/kholia/OSX-KVM for the scripts to download and setup
> the libvirt XML file. Finally the VMs are using "unsafe" disk-io mode for
> the disk images, so fsync and direct-io don't write through.
>
> I will install newer JDKs for macOS tomorrow to get the Lucene Main builds
> working again (the old macOS version did not support JDK>21).
>
> Uwe
>
> Am 19.03.2025 um 09:34 schrieb Dawid Weiss:
>
> The problem with Macos is the emulation.
>
> Kudos to you for even making it work. I never managed to get it working
> properly. There was always something that didn't work quite right or just
> broke after each Apple update. I'm sure
> they make it intentionally difficult to run a hackintosh.
>
> We do have true (I hope) mac runners on github so we do have some coverage
> there.
> https://github.com/apache/lucene/actions/runs/13932872683/job/38994065168
>
> D.
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi a small update,
the new virtualization works phantastic. Windows builds are a bit slower
than others, but this is caused by NTFS and Windows file locking.
Basically it is not different than with your local WIndows laptop. Macos
builds are nearly as fast as Linux builds (keep in mind that the number
of assigned CPUs differs between host and guest VM). This also applies
to Solr.
All JDKs were updated on Saturday. So Lucene main builds work.
Everything is using virtio drivers (network and disk images).
Interestingly, macOs BY DEFAULT ships with drivers for "virtio"
networking and disks (according to observations by others they do this
since Mojave)! So the whole thing is as fast as "plain hardware" and
real SSD disks (no SATA emulation needed):
@Paul Irwin: Because of macos shipping with "virtio drivers", they
support virtualization officially, also with KVM! They either test their
own builds in KVM for easier debugging the kernel or they officially
support it. The "osk" key is not a secret anymore. You could only argue
that "I" do not run it on native Apple hardware.
Uwe
Am 22.03.2025 um 00:14 schrieb Uwe Schindler:
> Hi,
>
> I converted "Windows VM" (that was easy) and reinstalled a brand-new
> "macOS 15 Sequoia VM" in Policeman Jenkins without using VirtualBOX
> anymore. All is running now as libvirt-based KVM instances. The builds
> are now
> faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaast!!!!!1!!1111!!!!!!!
> :-)
>
> The Windows and macOS VMs now build Lucene and especially Solr in
> similar times. I also extended disk space to 128 GiB per VM.
>
> I also have not seen any hangs of macOS, so the native support by KVM
> with some CPUID adaptions and good wrk by qemu is working well. Thanks
> to https://github.com/kholia/OSX-KVM for the scripts to download and
> setup the libvirt XML file. Finally the VMs are using "unsafe" disk-io
> mode for the disk images, so fsync and direct-io don't write through.
>
> I will install newer JDKs for macOS tomorrow to get the Lucene Main
> builds working again (the old macOS version did not support JDK>21).
>
> Uwe
>
> Am 19.03.2025 um 09:34 schrieb Dawid Weiss:
>>> The problem with Macos is the emulation.
>>>
>> Kudos to you for even making it work. I never managed to get it working
>> properly. There was always something that didn't work quite right or
>> just
>> broke after each Apple update. I'm sure
>> they make it intentionally difficult to run a hackintosh.
>>
>> We do have true (I hope) mac runners on github so we do have some
>> coverage
>> there.
>> https://github.com/apache/lucene/actions/runs/13932872683/job/38994065168
>>
>>
>> D.
>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I converted "Windows VM" (that was easy) and reinstalled a brand-new
"macOS 15 Sequoia VM" in Policeman Jenkins without using VirtualBOX
anymore. All is running now as libvirt-based KVM instances. The builds
are now
faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaast!!!!!1!!1111!!!!!!!
:-)
The Windows and macOS VMs now build Lucene and especially Solr in
similar times. I also extended disk space to 128 GiB per VM.
I also have not seen any hangs of macOS, so the native support by KVM
with some CPUID adaptions and good wrk by qemu is working well. Thanks
to https://github.com/kholia/OSX-KVM for the scripts to download and
setup the libvirt XML file. Finally the VMs are using "unsafe" disk-io
mode for the disk images, so fsync and direct-io don't write through.
I will install newer JDKs for macOS tomorrow to get the Lucene Main
builds working again (the old macOS version did not support JDK>21).
Uwe
Am 19.03.2025 um 09:34 schrieb Dawid Weiss:
>> The problem with Macos is the emulation.
>>
> Kudos to you for even making it work. I never managed to get it working
> properly. There was always something that didn't work quite right or just
> broke after each Apple update. I'm sure
> they make it intentionally difficult to run a hackintosh.
>
> We do have true (I hope) mac runners on github so we do have some coverage
> there.
> https://github.com/apache/lucene/actions/runs/13932872683/job/38994065168
>
> D.
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I converted "Windows VM" (that was easy) and reinstalled a brand-new
"macOS 15 Sequoia VM" in Policeman Jenkins without using VirtualBOX
anymore. All is running now as libvirt-based KVM instances. The builds
are now
faaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaast!!!!!1!!1111!!!!!!!
:-)
The Windows and macOS VMs now build Lucene and especially Solr in
similar times. I also extended disk space to 128 GiB per VM.
I also have not seen any hangs of macOS, so the native support by KVM
with some CPUID adaptions and good wrk by qemu is working well. Thanks
to https://github.com/kholia/OSX-KVM for the scripts to download and
setup the libvirt XML file. Finally the VMs are using "unsafe" disk-io
mode for the disk images, so fsync and direct-io don't write through.
I will install newer JDKs for macOS tomorrow to get the Lucene Main
builds working again (the old macOS version did not support JDK>21).
Uwe
Am 19.03.2025 um 09:34 schrieb Dawid Weiss:
>> The problem with Macos is the emulation.
>>
> Kudos to you for even making it work. I never managed to get it working
> properly. There was always something that didn't work quite right or just
> broke after each Apple update. I'm sure
> they make it intentionally difficult to run a hackintosh.
>
> We do have true (I hope) mac runners on github so we do have some coverage
> there.
> https://github.com/apache/lucene/actions/runs/13932872683/job/38994065168
>
> D.
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Policeman Jenkins => new hardware
Posted by Dawid Weiss <da...@gmail.com>.
>
> The problem with Macos is the emulation.
>
Kudos to you for even making it work. I never managed to get it working
properly. There was always something that didn't work quite right or just
broke after each Apple update. I'm sure
they make it intentionally difficult to run a hackintosh.
We do have true (I hope) mac runners on github so we do have some coverage
there.
https://github.com/apache/lucene/actions/runs/13932872683/job/38994065168
D.
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
It is same hardware. The problem with Macos is the emulation. Looks like the virtualization overhead is too large.
My hope is to get this working better with KVM. The problem with Macos is the processor hardware: if the CPU is not supported by Apple you have to hide some details by modifying CPU flags (see previous mail), so the osx kernel thinks its Intel hardware. It just refuses to boot otherwise.
KVM is not easily able to hide the real CPU identification.
Uwe
Am 18. März 2025 18:09:32 MEZ schrieb Houston Putman <ho...@apache.org>:
>Thanks for all the work here Uwe!
>
>I see that the OSX builds are a part of your TODOs, but with the new
>hardware, do you expect the OSX VM to be faster, or is the VM not living on
>the same hardware?
>We see a ton of OSX build failures because the "eventual consistency" in
>the tests doesn't expect the hardware to be quite as slow as the OSX VM
>is...
>
>- Houston
>
>On Tue, Mar 18, 2025 at 12:03 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> P.S.: Fun fact: The old policeman server's NVME SSDs were long after
>> their lifetime - so it was the main reason to replace it (the failing
>> network adaptor was just earlier). Lucene did a good job to burn the
>> SSDs. It is still interesting that Lucene/Solr's tests write more than
>> they read(!?!???!):
>>
>> # nvme smart-log /dev/nvme0
>> Smart Log for NVME device:nvme0 namespace-id:ffffffff
>> critical_warning : 0
>> temperature : 45 °C (318 K)
>> available_spare : 100%
>> available_spare_threshold : 10%
>> *percentage_used : 211%*
>> endurance group critical warning summary: 0
>> *Data Units Read : 317332814 (162.47 TB) Data Units Written : 2383037910
>> (1.22 PB)*
>> host_read_commands : 10452268853
>> host_write_commands : 57744004908
>> controller_busy_time : 73212
>> power_cycles : 9
>> power_on_hours : 45321
>> unsafe_shutdowns : 4
>> media_errors : 0
>> num_err_log_entries : 0
>> Warning Temperature Time : 0
>> Critical Composite Temperature Time : 0
>> Temperature Sensor 1 : 45 °C (318 K)
>> Thermal Management T1 Trans Count : 0
>> Thermal Management T2 Trans Count : 0
>> Thermal Management T1 Total Time : 0
>> Thermal Management T2 Total Time : 0
>>
>> # nvme smart-log /dev/nvme1
>> Smart Log for NVME device:nvme1 namespace-id:ffffffff
>> critical_warning : 0
>> temperature : 42 °C (315 K)
>> available_spare : 100%
>> available_spare_threshold : 10%*percentage_used : 217%
>> *endurance group critical warning summary: 0
>> *Data Units Read : 152984082 (78.33 TB) Data Units Written : 2385237910
>> (1.22 PB)*
>> host_read_commands : 1870329041
>> host_write_commands : 57743490085
>> controller_busy_time : 62644
>> power_cycles : 9
>> power_on_hours : 45321
>> unsafe_shutdowns : 4
>> media_errors : 0
>> num_err_log_entries : 0
>> Warning Temperature Time : 0
>> Critical Composite Temperature Time : 0
>> Temperature Sensor 1 : 42 °C (315 K)
>> Thermal Management T1 Trans Count : 0
>> Thermal Management T2 Trans Count : 0
>> Thermal Management T1 Total Time : 0
>> Thermal Management T2 Total Time : 0
>>
>> Uwe
>>
>> Am 18.03.2025 um 17:52 schrieb Uwe Schindler:
>> >
>> > Moin moin,
>> >
>> > Policeman Jenkins got new hardware yesterday - no functional changes.
>> >
>> > Background: The old server had some strange problems with the
>> > networking adaptor (Intel's "igb" kernel driver) about "Detected Tx
>> > Unit Hang". This caused some short downtimes and the monitoring
>> > complained all the time about lost pings which drove me crazy at
>> > weekend. It worked better after a restart and also with downgrade of
>> > kernel, but as I was about to replace the machine by a newer one, I
>> > ordered a replacement to new Hardware version (previously it was
>> > Hetzner AX51-NVME; now it is: Hetzner AX52).
>> >
>> > The migration was done starting yesterday lunch time europe (12:00
>> > CET) in the by booting the new server in the datacenter's recovery
>> > environment booted from network on both servers with a temporary IP
>> > and then mounting both root disks and doing a large rsync (with
>> > checksums, external attributes, numeric uid/gid and delete option).
>> > Luckily this worked with the old server (the Intel Adapter did not
>> > break). The whole downtime should have taken only 1 to 1.5 hours (the
>> > time copy with 1 GBits and reconfig needs), but unfortunately the
>> > PCIexpress on the new server complained about (recoverable) errors on
>> > the NVME communications. After some support roundtrips (they first
>> > replaced only the failing NVME controller which did not help), the
>> > replaced the whole server.
>> >
>> > At 18:30 CET, I started copy to new server again and all went well,
>> > dmesg showed no PCI express checksum errors. Finally, after fixing
>> > boot (the old server used MBR the new one EFI), the server was mounted
>> > at the original location by the team and all IPv4 adresses and IPv6
>> > network were available. Since then (approx 20:30 CET), Policeman
>> > Jenkins is back and running.
>> >
>> > The TODOs for the future:
>> >
>> > * Replace the MacOS VM and update it to a new version (it's
>> > complicated, as it is a "Hackintosh", so it shouldn't be there
>> > according to Apple)
>> > * Possibly migrate away from VirtualBOX to KVM, but it's unclear if
>> > Hackintoshs work there.
>> >
>> > Have fun with the new hardware, the builds on Lucene main branch are
>> > now 1.5 times faster (10 instead of 15 minutes).
>> >
>> > The new hardware is described here:
>> > https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
>> > let's see what comes out. No test failures yet.
>> >
>> > vendor_id : AuthenticAMD
>> > cpu family : 25
>> > model : 97
>> > model name : AMD Ryzen 7 7700 8-Core Processor
>> > stepping : 2
>> > microcode : 0xa601209
>> > cpu MHz : 5114.082
>> > cache size : 1024 KB
>> > physical id : 0
>> > siblings : 16
>> > core id : 7
>> > cpu cores : 8
>> > apicid : 15
>> > initial apicid : 15
>> > fpu : yes
>> > fpu_exception : yes
>> > cpuid level : 16
>> > wp : yes
>> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> > mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
>> > fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl
>> > xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
>> > monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
>> > rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
>> > 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb
>> > bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
>> > perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2
>> > smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap
>> > avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt
>> > xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
>> > cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru
>> > wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
>> > flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
>> > v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
>> > vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid
>> > overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
>> > bugs : sysret_ss_attrs spectre_v1 spectre_v2
>> > spec_store_bypass srso
>> > bogomips : 7585.28
>> > TLB size : 3584 4K pages
>> > clflush size : 64
>> > cache_alignment : 64
>> > address sizes : 48 bits physical, 48 bits virtual
>> > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>> >
>> > # lspci | fgrep -i volati
>> > 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
>> > NVMe SSD Controller PM9A1/PM9A3/980PRO
>> > 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400
>> > NVMe SSD [Hendrix]
>> >
>> > I have no idea why the replacement server has two different NVME SSDs,
>> > but you never know before what you get! From smart info I know that
>> > both SSDs were fresh (6 hours total uptime only).
>> >
>> > Uwe
>> >
>> > --
>> >
>> > Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail:uwe@thetaphi.de
>>
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Re: Policeman Jenkins => new hardware
Posted by Houston Putman <ho...@apache.org>.
Thanks for all the work here Uwe!
I see that the OSX builds are a part of your TODOs, but with the new
hardware, do you expect the OSX VM to be faster, or is the VM not living on
the same hardware?
We see a ton of OSX build failures because the "eventual consistency" in
the tests doesn't expect the hardware to be quite as slow as the OSX VM
is...
- Houston
On Tue, Mar 18, 2025 at 12:03 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> P.S.: Fun fact: The old policeman server's NVME SSDs were long after
> their lifetime - so it was the main reason to replace it (the failing
> network adaptor was just earlier). Lucene did a good job to burn the
> SSDs. It is still interesting that Lucene/Solr's tests write more than
> they read(!?!???!):
>
> # nvme smart-log /dev/nvme0
> Smart Log for NVME device:nvme0 namespace-id:ffffffff
> critical_warning : 0
> temperature : 45 °C (318 K)
> available_spare : 100%
> available_spare_threshold : 10%
> *percentage_used : 211%*
> endurance group critical warning summary: 0
> *Data Units Read : 317332814 (162.47 TB) Data Units Written : 2383037910
> (1.22 PB)*
> host_read_commands : 10452268853
> host_write_commands : 57744004908
> controller_busy_time : 73212
> power_cycles : 9
> power_on_hours : 45321
> unsafe_shutdowns : 4
> media_errors : 0
> num_err_log_entries : 0
> Warning Temperature Time : 0
> Critical Composite Temperature Time : 0
> Temperature Sensor 1 : 45 °C (318 K)
> Thermal Management T1 Trans Count : 0
> Thermal Management T2 Trans Count : 0
> Thermal Management T1 Total Time : 0
> Thermal Management T2 Total Time : 0
>
> # nvme smart-log /dev/nvme1
> Smart Log for NVME device:nvme1 namespace-id:ffffffff
> critical_warning : 0
> temperature : 42 °C (315 K)
> available_spare : 100%
> available_spare_threshold : 10%*percentage_used : 217%
> *endurance group critical warning summary: 0
> *Data Units Read : 152984082 (78.33 TB) Data Units Written : 2385237910
> (1.22 PB)*
> host_read_commands : 1870329041
> host_write_commands : 57743490085
> controller_busy_time : 62644
> power_cycles : 9
> power_on_hours : 45321
> unsafe_shutdowns : 4
> media_errors : 0
> num_err_log_entries : 0
> Warning Temperature Time : 0
> Critical Composite Temperature Time : 0
> Temperature Sensor 1 : 42 °C (315 K)
> Thermal Management T1 Trans Count : 0
> Thermal Management T2 Trans Count : 0
> Thermal Management T1 Total Time : 0
> Thermal Management T2 Total Time : 0
>
> Uwe
>
> Am 18.03.2025 um 17:52 schrieb Uwe Schindler:
> >
> > Moin moin,
> >
> > Policeman Jenkins got new hardware yesterday - no functional changes.
> >
> > Background: The old server had some strange problems with the
> > networking adaptor (Intel's "igb" kernel driver) about "Detected Tx
> > Unit Hang". This caused some short downtimes and the monitoring
> > complained all the time about lost pings which drove me crazy at
> > weekend. It worked better after a restart and also with downgrade of
> > kernel, but as I was about to replace the machine by a newer one, I
> > ordered a replacement to new Hardware version (previously it was
> > Hetzner AX51-NVME; now it is: Hetzner AX52).
> >
> > The migration was done starting yesterday lunch time europe (12:00
> > CET) in the by booting the new server in the datacenter's recovery
> > environment booted from network on both servers with a temporary IP
> > and then mounting both root disks and doing a large rsync (with
> > checksums, external attributes, numeric uid/gid and delete option).
> > Luckily this worked with the old server (the Intel Adapter did not
> > break). The whole downtime should have taken only 1 to 1.5 hours (the
> > time copy with 1 GBits and reconfig needs), but unfortunately the
> > PCIexpress on the new server complained about (recoverable) errors on
> > the NVME communications. After some support roundtrips (they first
> > replaced only the failing NVME controller which did not help), the
> > replaced the whole server.
> >
> > At 18:30 CET, I started copy to new server again and all went well,
> > dmesg showed no PCI express checksum errors. Finally, after fixing
> > boot (the old server used MBR the new one EFI), the server was mounted
> > at the original location by the team and all IPv4 adresses and IPv6
> > network were available. Since then (approx 20:30 CET), Policeman
> > Jenkins is back and running.
> >
> > The TODOs for the future:
> >
> > * Replace the MacOS VM and update it to a new version (it's
> > complicated, as it is a "Hackintosh", so it shouldn't be there
> > according to Apple)
> > * Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> > Hackintoshs work there.
> >
> > Have fun with the new hardware, the builds on Lucene main branch are
> > now 1.5 times faster (10 instead of 15 minutes).
> >
> > The new hardware is described here:
> > https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
> > let's see what comes out. No test failures yet.
> >
> > vendor_id : AuthenticAMD
> > cpu family : 25
> > model : 97
> > model name : AMD Ryzen 7 7700 8-Core Processor
> > stepping : 2
> > microcode : 0xa601209
> > cpu MHz : 5114.082
> > cache size : 1024 KB
> > physical id : 0
> > siblings : 16
> > core id : 7
> > cpu cores : 8
> > apicid : 15
> > initial apicid : 15
> > fpu : yes
> > fpu_exception : yes
> > cpuid level : 16
> > wp : yes
> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> > mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> > fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl
> > xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
> > monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
> > rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
> > 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb
> > bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
> > perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2
> > smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap
> > avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt
> > xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
> > cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru
> > wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
> > flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
> > v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
> > vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid
> > overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> > bugs : sysret_ss_attrs spectre_v1 spectre_v2
> > spec_store_bypass srso
> > bogomips : 7585.28
> > TLB size : 3584 4K pages
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 48 bits physical, 48 bits virtual
> > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> >
> > # lspci | fgrep -i volati
> > 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
> > NVMe SSD Controller PM9A1/PM9A3/980PRO
> > 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400
> > NVMe SSD [Hendrix]
> >
> > I have no idea why the replacement server has two different NVME SSDs,
> > but you never know before what you get! From smart info I know that
> > both SSDs were fresh (6 hours total uptime only).
> >
> > Uwe
> >
> > --
> >
> > Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
> > eMail: uwe@thetaphi.de
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail:uwe@thetaphi.de
>
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
P.S.: Fun fact: The old policeman server's NVME SSDs were long after
their lifetime - so it was the main reason to replace it (the failing
network adaptor was just earlier). Lucene did a good job to burn the
SSDs. It is still interesting that Lucene/Solr's tests write more than
they read(!?!???!):
# nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : 45 °C (318 K)
available_spare : 100%
available_spare_threshold : 10%
*percentage_used : 211%*
endurance group critical warning summary: 0
*Data Units Read : 317332814 (162.47 TB) Data Units Written : 2383037910
(1.22 PB)*
host_read_commands : 10452268853
host_write_commands : 57744004908
controller_busy_time : 73212
power_cycles : 9
power_on_hours : 45321
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 45 °C (318 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning : 0
temperature : 42 °C (315 K)
available_spare : 100%
available_spare_threshold : 10%*percentage_used : 217% *endurance group critical warning summary: 0
*Data Units Read : 152984082 (78.33 TB) Data Units Written : 2385237910
(1.22 PB)*
host_read_commands : 1870329041
host_write_commands : 57743490085
controller_busy_time : 62644
power_cycles : 9
power_on_hours : 45321
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 42 °C (315 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
Uwe
Am 18.03.2025 um 17:52 schrieb Uwe Schindler:
>
> Moin moin,
>
> Policeman Jenkins got new hardware yesterday - no functional changes.
>
> Background: The old server had some strange problems with the
> networking adaptor (Intel's "igb" kernel driver) about "Detected Tx
> Unit Hang". This caused some short downtimes and the monitoring
> complained all the time about lost pings which drove me crazy at
> weekend. It worked better after a restart and also with downgrade of
> kernel, but as I was about to replace the machine by a newer one, I
> ordered a replacement to new Hardware version (previously it was
> Hetzner AX51-NVME; now it is: Hetzner AX52).
>
> The migration was done starting yesterday lunch time europe (12:00
> CET) in the by booting the new server in the datacenter's recovery
> environment booted from network on both servers with a temporary IP
> and then mounting both root disks and doing a large rsync (with
> checksums, external attributes, numeric uid/gid and delete option).
> Luckily this worked with the old server (the Intel Adapter did not
> break). The whole downtime should have taken only 1 to 1.5 hours (the
> time copy with 1 GBits and reconfig needs), but unfortunately the
> PCIexpress on the new server complained about (recoverable) errors on
> the NVME communications. After some support roundtrips (they first
> replaced only the failing NVME controller which did not help), the
> replaced the whole server.
>
> At 18:30 CET, I started copy to new server again and all went well,
> dmesg showed no PCI express checksum errors. Finally, after fixing
> boot (the old server used MBR the new one EFI), the server was mounted
> at the original location by the team and all IPv4 adresses and IPv6
> network were available. Since then (approx 20:30 CET), Policeman
> Jenkins is back and running.
>
> The TODOs for the future:
>
> * Replace the MacOS VM and update it to a new version (it's
> complicated, as it is a "Hackintosh", so it shouldn't be there
> according to Apple)
> * Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
>
> Have fun with the new hardware, the builds on Lucene main branch are
> now 1.5 times faster (10 instead of 15 minutes).
>
> The new hardware is described here:
> https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
> let's see what comes out. No test failures yet.
>
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 97
> model name : AMD Ryzen 7 7700 8-Core Processor
> stepping : 2
> microcode : 0xa601209
> cpu MHz : 5114.082
> cache size : 1024 KB
> physical id : 0
> siblings : 16
> core id : 7
> cpu cores : 8
> apicid : 15
> initial apicid : 15
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl
> xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
> monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
> rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
> 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb
> bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
> perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2
> smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap
> avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt
> xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
> cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru
> wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
> flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
> v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
> vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid
> overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> bugs : sysret_ss_attrs spectre_v1 spectre_v2
> spec_store_bypass srso
> bogomips : 7585.28
> TLB size : 3584 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> # lspci | fgrep -i volati
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
> NVMe SSD Controller PM9A1/PM9A3/980PRO
> 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400
> NVMe SSD [Hendrix]
>
> I have no idea why the replacement server has two different NVME SSDs,
> but you never know before what you get! From smart info I know that
> both SSDs were fresh (6 hours total uptime only).
>
> Uwe
>
> --
>
> Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
> eMail: uwe@thetaphi.de
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Paul,
as Robert already said, the jenkins.thetaphi.de server is hosted by an
individual (myself), so the ASF is not affected by that. It offers a
Jenkins server with Linux and a Windows VM and a macOS VM. I know the
discussion how legal that is. The whole setup runs like this since very
long time, so it is not new.
If Apple wants to fight against me, they should go ahead. I can for sure
switch off the VM in worst case, but using a publicly available string
to feed the Apple-SMC with a variable thats read by the Darwin Kernel on
bootup is not a violation. BUT: On the long term, the VM is questionable
anyways, as Apple moved away from x86-64, so indeed a Github runner is
better. I think we use one for Lucene already, Dawid knows better.
Uwe
Am 19.03.2025 um 16:19 schrieb Paul Irwin:
> First, congrats on setting up the new hardware! It's amazing what modern
> hardware is capable of for such a low cost.
>
> It seems unwise to continue using a Hackintosh VM. It is questionably legal
> at best (assuming a license was bought), and could result in consequences
> for either the ASF or individuals involved in setting that up. The "osk"
> string mentioned below is designed to be a warning about this. I don't mean
> to kick an anthill, but I searched the archives for this list and haven't
> seen that discussed until now. (I also don't think it's productive to
> debate the merits of Apple's licensing choices or their walled garden; we
> can't change that or the possible consequences either way.)
>
> There are several legal options available for running macOS builds/jobs.
> One is GitHub Actions, where the job could be manually dispatched via API
> from Jenkins if needed.
> https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
>
> There's also MacinCloud that offers hosted Mac servers for far less than
> equivalently spec'ed Linux Azure/AWS/GCP VMs. https://www.macincloud.com/
>
> I also daily drive macOS, and if there's any way I can help the team by
> testing issues/fixes/features/etc. on macOS, just let me know and I'd be
> happy to help on my end.
>
> Paul Irwin
>
>
> On Wed, Mar 19, 2025 at 8:22 AM Robert Muir <rc...@gmail.com> wrote:
>
>> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>>> Possibly migrate away from VirtualBOX to KVM, but it's unclear if
>> Hackintoshs work there.
>> -device
>> isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: uwe@thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Policeman Jenkins => new hardware
Posted by Robert Muir <rc...@gmail.com>.
Nothing to do with asf. This is just one individual setting up a Jenkins.
Anyone can do that.
On Wed, Mar 19, 2025, 11:20 AM Paul Irwin <pa...@gmail.com> wrote:
> First, congrats on setting up the new hardware! It's amazing what modern
> hardware is capable of for such a low cost.
>
> It seems unwise to continue using a Hackintosh VM. It is questionably
> legal at best (assuming a license was bought), and could result in
> consequences for either the ASF or individuals involved in setting that up.
> The "osk" string mentioned below is designed to be a warning about this. I
> don't mean to kick an anthill, but I searched the archives for this list
> and haven't seen that discussed until now. (I also don't think it's
> productive to debate the merits of Apple's licensing choices or their
> walled garden; we can't change that or the possible consequences either
> way.)
>
> There are several legal options available for running macOS builds/jobs.
> One is GitHub Actions, where the job could be manually dispatched via API
> from Jenkins if needed.
> https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
>
> There's also MacinCloud that offers hosted Mac servers for far less than
> equivalently spec'ed Linux Azure/AWS/GCP VMs. https://www.macincloud.com/
>
> I also daily drive macOS, and if there's any way I can help the team by
> testing issues/fixes/features/etc. on macOS, just let me know and I'd be
> happy to help on my end.
>
> Paul Irwin
>
>
> On Wed, Mar 19, 2025 at 8:22 AM Robert Muir <rc...@gmail.com> wrote:
>
>> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
>> Hackintoshs work there.
>> >
>>
>> -device
>> isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Policeman Jenkins => new hardware
Posted by Robert Muir <rc...@gmail.com>.
Nothing to do with asf. This is just one individual setting up a Jenkins.
Anyone can do that.
On Wed, Mar 19, 2025, 11:20 AM Paul Irwin <pa...@gmail.com> wrote:
> First, congrats on setting up the new hardware! It's amazing what modern
> hardware is capable of for such a low cost.
>
> It seems unwise to continue using a Hackintosh VM. It is questionably
> legal at best (assuming a license was bought), and could result in
> consequences for either the ASF or individuals involved in setting that up.
> The "osk" string mentioned below is designed to be a warning about this. I
> don't mean to kick an anthill, but I searched the archives for this list
> and haven't seen that discussed until now. (I also don't think it's
> productive to debate the merits of Apple's licensing choices or their
> walled garden; we can't change that or the possible consequences either
> way.)
>
> There are several legal options available for running macOS builds/jobs.
> One is GitHub Actions, where the job could be manually dispatched via API
> from Jenkins if needed.
> https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
>
> There's also MacinCloud that offers hosted Mac servers for far less than
> equivalently spec'ed Linux Azure/AWS/GCP VMs. https://www.macincloud.com/
>
> I also daily drive macOS, and if there's any way I can help the team by
> testing issues/fixes/features/etc. on macOS, just let me know and I'd be
> happy to help on my end.
>
> Paul Irwin
>
>
> On Wed, Mar 19, 2025 at 8:22 AM Robert Muir <rc...@gmail.com> wrote:
>
>> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>>
>> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
>> Hackintoshs work there.
>> >
>>
>> -device
>> isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: Policeman Jenkins => new hardware
Posted by Paul Irwin <pa...@gmail.com>.
First, congrats on setting up the new hardware! It's amazing what modern
hardware is capable of for such a low cost.
It seems unwise to continue using a Hackintosh VM. It is questionably legal
at best (assuming a license was bought), and could result in consequences
for either the ASF or individuals involved in setting that up. The "osk"
string mentioned below is designed to be a warning about this. I don't mean
to kick an anthill, but I searched the archives for this list and haven't
seen that discussed until now. (I also don't think it's productive to
debate the merits of Apple's licensing choices or their walled garden; we
can't change that or the possible consequences either way.)
There are several legal options available for running macOS builds/jobs.
One is GitHub Actions, where the job could be manually dispatched via API
from Jenkins if needed.
https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
There's also MacinCloud that offers hosted Mac servers for far less than
equivalently spec'ed Linux Azure/AWS/GCP VMs. https://www.macincloud.com/
I also daily drive macOS, and if there's any way I can help the team by
testing issues/fixes/features/etc. on macOS, just let me know and I'd be
happy to help on my end.
Paul Irwin
On Wed, Mar 19, 2025 at 8:22 AM Robert Muir <rc...@gmail.com> wrote:
> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
> >
>
> -device
> isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Policeman Jenkins => new hardware
Posted by Paul Irwin <pa...@gmail.com>.
First, congrats on setting up the new hardware! It's amazing what modern
hardware is capable of for such a low cost.
It seems unwise to continue using a Hackintosh VM. It is questionably legal
at best (assuming a license was bought), and could result in consequences
for either the ASF or individuals involved in setting that up. The "osk"
string mentioned below is designed to be a warning about this. I don't mean
to kick an anthill, but I searched the archives for this list and haven't
seen that discussed until now. (I also don't think it's productive to
debate the merits of Apple's licensing choices or their walled garden; we
can't change that or the possible consequences either way.)
There are several legal options available for running macOS builds/jobs.
One is GitHub Actions, where the job could be manually dispatched via API
from Jenkins if needed.
https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event
There's also MacinCloud that offers hosted Mac servers for far less than
equivalently spec'ed Linux Azure/AWS/GCP VMs. https://www.macincloud.com/
I also daily drive macOS, and if there's any way I can help the team by
testing issues/fixes/features/etc. on macOS, just let me know and I'd be
happy to help on my end.
Paul Irwin
On Wed, Mar 19, 2025 at 8:22 AM Robert Muir <rc...@gmail.com> wrote:
> On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
> >
>
> -device
> isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Policeman Jenkins => new hardware
Posted by Robert Muir <rc...@gmail.com>.
On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> Possibly migrate away from VirtualBOX to KVM, but it's unclear if Hackintoshs work there.
>
-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Policeman Jenkins => new hardware
Posted by Uwe Schindler <uw...@thetaphi.de>.
P.S.: Fun fact: The old policeman server's NVME SSDs were long after
their lifetime - so it was the main reason to replace it (the failing
network adaptor was just earlier). Lucene did a good job to burn the
SSDs. It is still interesting that Lucene/Solr's tests write more than
they read(!?!???!):
# nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : 45 °C (318 K)
available_spare : 100%
available_spare_threshold : 10%
*percentage_used : 211%*
endurance group critical warning summary: 0
*Data Units Read : 317332814 (162.47 TB) Data Units Written : 2383037910
(1.22 PB)*
host_read_commands : 10452268853
host_write_commands : 57744004908
controller_busy_time : 73212
power_cycles : 9
power_on_hours : 45321
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 45 °C (318 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
# nvme smart-log /dev/nvme1
Smart Log for NVME device:nvme1 namespace-id:ffffffff
critical_warning : 0
temperature : 42 °C (315 K)
available_spare : 100%
available_spare_threshold : 10%*percentage_used : 217% *endurance group critical warning summary: 0
*Data Units Read : 152984082 (78.33 TB) Data Units Written : 2385237910
(1.22 PB)*
host_read_commands : 1870329041
host_write_commands : 57743490085
controller_busy_time : 62644
power_cycles : 9
power_on_hours : 45321
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 42 °C (315 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0
Uwe
Am 18.03.2025 um 17:52 schrieb Uwe Schindler:
>
> Moin moin,
>
> Policeman Jenkins got new hardware yesterday - no functional changes.
>
> Background: The old server had some strange problems with the
> networking adaptor (Intel's "igb" kernel driver) about "Detected Tx
> Unit Hang". This caused some short downtimes and the monitoring
> complained all the time about lost pings which drove me crazy at
> weekend. It worked better after a restart and also with downgrade of
> kernel, but as I was about to replace the machine by a newer one, I
> ordered a replacement to new Hardware version (previously it was
> Hetzner AX51-NVME; now it is: Hetzner AX52).
>
> The migration was done starting yesterday lunch time europe (12:00
> CET) in the by booting the new server in the datacenter's recovery
> environment booted from network on both servers with a temporary IP
> and then mounting both root disks and doing a large rsync (with
> checksums, external attributes, numeric uid/gid and delete option).
> Luckily this worked with the old server (the Intel Adapter did not
> break). The whole downtime should have taken only 1 to 1.5 hours (the
> time copy with 1 GBits and reconfig needs), but unfortunately the
> PCIexpress on the new server complained about (recoverable) errors on
> the NVME communications. After some support roundtrips (they first
> replaced only the failing NVME controller which did not help), the
> replaced the whole server.
>
> At 18:30 CET, I started copy to new server again and all went well,
> dmesg showed no PCI express checksum errors. Finally, after fixing
> boot (the old server used MBR the new one EFI), the server was mounted
> at the original location by the team and all IPv4 adresses and IPv6
> network were available. Since then (approx 20:30 CET), Policeman
> Jenkins is back and running.
>
> The TODOs for the future:
>
> * Replace the MacOS VM and update it to a new version (it's
> complicated, as it is a "Hackintosh", so it shouldn't be there
> according to Apple)
> * Possibly migrate away from VirtualBOX to KVM, but it's unclear if
> Hackintoshs work there.
>
> Have fun with the new hardware, the builds on Lucene main branch are
> now 1.5 times faster (10 instead of 15 minutes).
>
> The new hardware is described here:
> https://www.hetzner.com/dedicated-rootserver/ax52/; it has AVX 512....
> let's see what comes out. No test failures yet.
>
> vendor_id : AuthenticAMD
> cpu family : 25
> model : 97
> model name : AMD Ryzen 7 7700 8-Core Processor
> stepping : 2
> microcode : 0xa601209
> cpu MHz : 5114.082
> cache size : 1024 KB
> physical id : 0
> siblings : 16
> core id : 7
> cpu cores : 8
> apicid : 15
> initial apicid : 15
> fpu : yes
> fpu_exception : yes
> cpuid level : 16
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
> fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl
> xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq
> monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
> rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
> 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb
> bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
> perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2
> smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap
> avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt
> xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
> cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru
> wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean
> flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic
> v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes
> vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid
> overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
> bugs : sysret_ss_attrs spectre_v1 spectre_v2
> spec_store_bypass srso
> bogomips : 7585.28
> TLB size : 3584 4K pages
> clflush size : 64
> cache_alignment : 64
> address sizes : 48 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
>
> # lspci | fgrep -i volati
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
> NVMe SSD Controller PM9A1/PM9A3/980PRO
> 02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400
> NVMe SSD [Hendrix]
>
> I have no idea why the replacement server has two different NVME SSDs,
> but you never know before what you get! From smart info I know that
> both SSDs were fresh (6 hours total uptime only).
>
> Uwe
>
> --
>
> Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de
> eMail: uwe@thetaphi.de
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de
Re: Policeman Jenkins => new hardware
Posted by Robert Muir <rc...@gmail.com>.
On Tue, Mar 18, 2025 at 12:53 PM Uwe Schindler <uw...@thetaphi.de> wrote:
> Possibly migrate away from VirtualBOX to KVM, but it's unclear if Hackintoshs work there.
>
-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org