Did some work on a REL7 server earlier today (started at 5:30 am!)
Nightmarish scenario where someone whose role is titled “architect” migrates application stacks by jiggling IP addresses and hostnames around.
e.g.
ServerB is in test and IP address 10.10.10.28
replacing
ServerA, currently in prod and IP address 10.10.10.21 - but being replaced by ServerB.
We allocated an unused IP address (10.10.10.27) for ServerA, and update it with unused hostname :
ServerA (REL7) becomes ServerA-old with IP address 10.10.10.27. And this is redhat so our NIC settings are in /etc/sysconfig/network-scripts/ifcfg-ens192
e.g.
cp /etc/sysconfig/network-scripts/ifcfg-ens192 /etc/sysconfig/network-scripts/ifcfg-ens192-ORIG
vi /etc/sysconfig/network-scripts/ifcfg-ens192
set hostname in /etc/hostname, reboot.
ServerB (REL8) becomes ServerA with IP address 10.10.10.21!
ServerA-old boots up and when we try to get to 10.10.10.21 from the old renamed server “ServerA-old” - and - it’s going to itself!
Doh - took about half hour to figure out, networkd (or whatever it’s called in RPM world) was reading BOTH ifcfg-ens192 - and - ifcfg-ens192-ORIG and “multihoming” itself, i.e. with the new, and the old, IP address! Doh!
Lesson to self : sure, by all means, make backups of crucial config files, but put them in DIFFERENT FOLDERS!
I already had a go at them last week telling them how lousy this was (switcheroo on hostnames and IP addresses) - and we were going to run it “properly” where new features are added to different hostnames and IP addresses, but someone decided it was too late to change, and they hadn’t done that in “test” (I wasn’t involved in test - project was thrown at me at the last minute).
Next time (planned for next ~12 months) round, it’s going to be architected properly, so we standup ALL new servers, new servernames, new IP addresses (these are private addresses so we’ll never run out!), and the only major changes will probably be DNS A records and CNAMES! No servers will be renamed, no IP addresses will be “recycled”, and only a few electrons will be harmed during the making of…