I am in no way trying to poopoo the work done by the author of this post. I really like when people come out and show what they found and how they fixed things. I also didn’t attempt to contact the author as I’m sure he’s been inundated with emails/comments on what he woulda/shoulda/coulda done. My excuse is I was up one evening and couldn’t sleep. As such I was catching up on some reading (no downtime, ever) and this was one of the articles I read. I thought I’d go through the various comments and offer my rather uneducated opinions…
The blog post in question.
Jan 7 15:05:31 asterisk userhelper: running ‘/sbin/reboot -f’ with root privileges on behalf of ‘root’
At this point, I’d assume that the entire server has been compromised. This is the point I would have (if possible), shut the server down and move to a DR site or backup server to provide the services required. The fact that someone other than yourself or one of your trusted sys-admins has rebooted the server from the root account means you no longer control the server. From here I would have started by looking at the various shell history files to see what else had been run.
I’m a little curious how the attackers got root or even system level access to the system by simply “hacking” the Zap/Skype trunks available on the system. In my fairly limited experience in VOIP compromises, the attackers have mostly found loopholes in the dial plan and used these loopholes to make international calls.
Next, I decided to check the /etc/password file to see if the hacker created any backdoor user accounts. I didn’t see any, however I did notice that nobody account had this:
Here’s a good example of where something like OSSEC would have come in handy. OSSEC would have notified the sys-admins of the change to the system accounts as it happened (IF the mail system notification was configured and not disabled by the attacker).
So if the ‘nobody’ account is able to get bash-level access, they can view and modify any Asterisk file! That means access to the SIP passwords and the ability to launch the Asterisk CLI! From the Asterisk CLI you can do just about anything, including make outbound calls or even delete critical Asterisk files.
I find this hard to believe. Then again I haven’t played with an Asterisk installation in ages. I’m a big fan of running services under their own account. This would have mitigated this issue somewhat. The attacker would have had to have changed the shell of the user running the Asterisk service.
Looking at the ‘history’ command output for user ‘root’ I saw these two suspicious commands executed:
333 su aster
334 su nobody
This is another area where OSSEC would have saved the day with its watcher service and email notification. The su action would have triggered an alert to the sys-admins notifying them of the action. Hopefully the sys-admin would have realized something was up and done a little digging.
Surprisingly, the hacker didn’t clean up their history. Or if they did, they forgot to clean up these two commands. I don’t have an ‘aster’ account, so that command fails, but obviously the ‘su nobody’ works, causing the root user to login as ‘nobody’. The beauty of using the ‘nobody’ account is that all the commands executed by the hacker doesn’t show up in the ‘root’ admin account that most IT admins monitor. At that point I could no longer see the commands the hacker was executing under the ‘nobody’ bash shell. Even when I logged in as ‘nobody’ (after changing password) I couldn’t see any history.
There are a number of ways to ship bash command history off to a remote syslog server. These logs should be reviewed regularly. That said, this is probably a fairly difficult thing to do if you’re a small IT shop. A properly configured SEIM installation with alerts for commands being run by strange or system users would have caught this as well.
Interestingly, the 1.call file the hacker chose to do was very simplistic. It didn’t do anything other than dial the number and then hang around for 36000s doing nothing. No prompts played or anything. Obviously, if someone answered the call, they’d likely hang up in 5-10s. It seems as though the hacker was making prank calls, since their script didn’t do anything. What sort of motive is that?
My only guess here is it’s probably some kind of premium rate number for incoming calls. I’ve seen this happen locally, along with the “free international calls” hacks in the past. It’s unfortunate that people/hackers are no longer interested in simply showing their skills, it’s all about the money now and if this was anything other than a premium rate scam of some kind, I’d be very surprised.
- Never allow SSH from the outside, even though it is convenient to remotely administer that way. If you MUST have SSH open to the outside, you should consider using Fail2Ban, which blocks IP addresses . It even supportsblocking hackers brute-force attacking your SIP credentials should you have port 5060 open to the Internet for remote phones.
- Make sure you regularly check your /etc/passwd and /etc/shadow to ensure no new accounts and that accounts don’t have /sbin/bash access (except those you know about, such as ‘root’)
- Use LogWatch to email you daily reports of what’s happening on your Asterisk box, such as logins, failed SSH attempts, etc.
1. If only life and system administration were that simple. In the real world, this isn’t really an easy one to sell. Personally I change what port SSH listens on along with a firewall burst rate limiter and fail2ban. That along with SSH key authentication instead of simple password authentication should be enough.
2. OSSEC file watcher with email notification should resolve this issue quite nicely. It’ll also alert the sys-admin should new packages be installed etc..
3. While daily reports are great, I’ve found that if you receive them daily and they look very similar, you will tend to skim read them and eventually ignore them altogether. A better option would be to be emailed only when things look wrong or change unexpectedly.
I’m not 100% sure yet if there is still a rootkit on the Linux box. Although there are anti-virus utilities for Linux, I am loathe to put anti-virus on a production PBX if it stays resident in memory and uses CPU cycles.
He’s absolutely right here. I’d go with either rkhunter or chkrootkit here. Both of these tools can be run and updated from Cron which would mean little interaction from busy sys-admins. Also the lack of a daily report from either tool could be a red flag that something is up…
Found this interesting error in the logs:
zdump: error: Bind to port 10001 on 0.0.0.0 failed: Address already in use.
This should have been stopped by a local firewall running on the system. I had a system once that was compromised via SSH (a story for another time) and the attackers had dropped a SSH scanner which didn’t work on account of the egress filtering in place on the system. Similarly this attack should have been stopped by a firewall filtering incoming connections to “unnecessary” or unknown services.
Looking through to comments I see many people mention OSSEC and Rkhunter which makes me feel a little warm and fuzzy inside. I know this is all probably a little redundant but it was on my mind. I also know that securing systems may be on the back burner for many system administrators who are not only pressed for time, but also money. I’ve been in that boat a few times. These days it’s difficult to keep on top of systems and their state of (in)security which makes a proper monitoring system all the more important, even if it’s a simple homegrown solution to begin with.