After updating the SSL certificate on our Splunk servers, I needed to verify that the new SSL certificate was in place and working. I did this by browsing to each server on port 8089.
I was able to connect to all of the servers except one:
To check that Splunk was listening on the correct port, I ran splunk show splunkd-port on the affected server. This timed out with an error:
Next I checked the port using netstat:
CLOSE_WAIT indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. The netstat information didn’t help me much, but the splunk command clearly showed an issue within Splunk.
The next logical step was to check for errors in the splunkd log:
I did a quick timechart of the errors and noticed they started occurring directly after the SSL cert change. Bingo!
I checked the SSL configuration in server.conf, and sure enough there was a typo in the sslRootCAPath. After correcting the typo and restarting Splunk, I was able to browse to port 8089 on the server.
In this case the troubleshooting was very straight forward. As I had just updated the SSL certificates, I knew exactly where to look for the root cause. I figure it’s still worth documenting since the troubleshooting steps might be helpful to others.