For such a unix user, ssh is a commonly used application which provide some kind of remote access to some servers. I’ve used it since my first experienced using unix based OS computer. And from day on, I’ve never faced any serious problems while using it.
But late at yesterday’s nite, this stuff was starting disturbing me. I’ve just reconfigured one of my experimental server, an old hp-compaq proliant ML370 series running Debian operating system. Evertyhing runs normally before. It was in a NAT networks, but later I would like to move it directly in to my campus networks using public IP. And the problem arises right after I change its IP parameters and restart the machine. It’s no longer accessible by remote access. The machine is alive, ping results do well.
But wait a minute, there’re some more anomalies I found. I suddenly can log in to the machine using the same ssh method! *what the…* For not wasting time, I reconfigure sshd server configuration file, check all networks configuration files too. But suddenly my connection was stopped. *now what…* I try to reconnect; insert the username, and the server responses by gives challenge for related password. I wrote down the password, but the machine rejected it, said “wrong password..” *hello…???* I retried three times, but no positive result I got.
Then, I walk to the server room, log in to it directly. I recheck all network configuration once more, to make sure that there’s no such thing I’ve missed. But everything seems okay. I do ssh connection to another server from this machine, passed. And do ssh again from that machine back to the one that arising problems, it also passed. Hmmm…, how weird. I look at the clock, it’s already 23.40. D*mn!
And the next 2 hours were filled with google-related activity.
It feels like I was in my daily works in the office, do some kind of Proof of Concept, PoC. And the sub theme is Root Cause Analysis. But different from the ordinary, this session was held in sleepy mode.
*well, how can I show my sleepy mode face in front of my manager in daily works?*
In my point of view, the main problem is (by analyse syslog) that the machine periodically changes it DSA key. It’s weird since one machine with one IP address (and related mac address) would only generate one DSA key. This key would be accept by the opponent (in this case ssh client) and saved. It is describe more clearly while using debug option in ssh connection. My early asumption is that it related to some ssh configuration or firewall issue.
After randomly followed some instructions and methods from some webs, finally I decided to temporarily finished my works. Enough for tonight, I thought. Let’s bring them all to my dreams, and hope by tomorrow morning when I wake up, I’ve got some enlightment… *silly thought*
In fact, even wake up late in the morning didn’t make any enlightment nor improvement.
With my typical lazy move, I woke up, check my e-mail first, and during that, I try to arrange my time. Beside of this, I have to do another software test related to my company job works. Take a bath for a while and after that, I start to continue my pending works yesterday.
Another two hours was spent, until I realized one thing (that would be the ultimate hint). I did nslookup , and get shocked after read the result. My server IP’s has two different host name listed in the DNS server. Well, no wonder it could happened. One IP is registered to two different name and different machine. That’s the only reason why the DSA key periodically changes. In one time, the first machine is taking action. But in the other time,the other machine is. By this, I also state that it is the main reason why in the syslog, it sounds something related to ip_spoofing. Yeah… what else?
I could not remember how this silly thing can occur. I mean, there’re two host name listed in the nameserver for a same IP? Well.., my the DNS admin did some fault. After change the host’s IP address, ssh service then run normally. I also report this situation to the DNS admin, one of my campus colleague, and ask him to reconfigure the list.
Anyway, it’s my first writing about some technical stuff.