server2 requires manual config after power cycle #9
Description
server2 was responsive to ping but unresponsive to SSH or HTTP. We went to Fremont to manually power cycle it. This fixed ssh, sympa, phpmyadmin, and phppgadmin, but not postfix or blog.
using salt-ssh 'server2' state.highstate was not showing output until forced to quit with control-c, which, if allowed to run long enough, did show the normal output. After fixing the routes, this ran normally and showed output on its own.
Mailserver
Nullmailer was showing temporary error in name resolution for connecting to mail.cloyne.org. This suggested either a DNS issue or an issue with postfix. First, I tried to test mail.cloyne.org using telnet. This worked from my laptop but failed from server2, so that meant it wasn't an issue with mail.cloyne.org. Next, I checked /etc/resolv.conf to find the IP of our DNS server and then tried pinging it. Ping failed.
I re-added the routes we added when we first moved the servers:
ip route add 64.62.133.41 dev [eth0 for server2, p1p1 for server3]
ip route replace default via 64.62.133.41 dev [eth0 for server2, p1p1 for server3] src 64.62.133.[44 for server2, 45 for server3]
This fixed the mailserver but not the blog.
Blog
After fixing the route, I ran salt-ssh server2 state.highstate again with just mysql and its dependents and this fixed it. Probably because it upgraded the tozd images which had been upgraded? Not sure what was wrong.