What To Do When Your Web Server Gets Hacked

This question keeps being asked repeatedly by the victims of hackers breaking into their web server. The answers very rarely change, but people keep asking the question. I’m not sure why. Perhaps people just don’t like the answers they’ve seen when searching for help, or they can’t find someone they trust to give them advice. Or perhaps people read an answer to this question and focus too much on the 5% of why their case is special and different from the answers they can find online and miss the 95% of the question and answer where their case is near enough the same as the one they read online.

That brings me to my first important nugget of information. I really do appreciate that you are a special unique snowflake. I appreciate that your website is too, as it’s a reflection of you and your business or at the very least, your hard work on behalf of an employer. But to someone on the outside looking in, whether a computer security person looking at the problem to try and help you or even the attacker himself, it is very likely that your problem will be at least 95% identical to every other case they’ve ever looked at.

Don’t take the attack personally, and don’t take the recommendations that follow here or that you get from other people personally. If you are reading this after just becoming the victim of a website hack then I really am sorry, and I really hope you can find something helpful here, but this is not the time to let your ego get in the way of what you need to do.
You have just found out that your server(s) got hacked. Now what?

Do not panic. Absolutely do not act in haste, and absolutely do not try and pretend things never happened and not act at all.

If you’ve read my previous post about risk management, or any of the much better articles about risk management out on the web, you’ll understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact.

Some of these steps are going to hurt, and (unless your website holds a copy of my details) I really don’t care if you ignore all or some of these steps but doing so will make things better in the end. The medicine might taste awful but sometimes you have to overlook that if you really want the cure to work.

Stop the problem from becoming worse than it already is:

  1. The first thing you should do is disconnect the affected systems from the Internet. Whatever other problems you have, leaving the system connected to the web will only allow the attack to continue. I mean this quite literally; get someone to physically visit the server and unplug network cables if that is what it takes, but disconnect the victim from its muggers before you try to do anything else.
  2. Change all your passwords for all accounts on all computers that are on the same network as the compromised systems. No really. All accounts. All computers. Yes, you’re right, this might be overkill; on the other hand, it might not. You don’t know either way, do you?
  3. Check your other systems. Pay special attention to other Internet facing services, and to those that hold financial or other commercially sensitive data.
  4. If the system holds anyone’s personal data, make a full and frank disclosure to anyone potentially affected at once. I know this one is tough. I know this one is going to hurt. I know that many businesses want to sweep this kind of problem under the carpet but I’m afraid you’re just going to have to deal with it.

Still hesitating to take this last step? I understand, I do. Nevertheless, let me break this down for you Robert-style:

In some places you might well have a legal requirement to inform the authorities and/or the victims of this kind of privacy breach. However annoyed your customers might be to have you tell them about a problem, they’ll be far more annoyed if you don’t tell them, and they only find out for themselves after someone charges $8,000 worth of goods using the credit card details they stole from your site.

Remember what I said in the previous section? The bad thing has already happened. The only question now is how well you deal with it.


How To Integrate Modern Macs To A Windows Network

A walk through of the steps needed to join an OS X 10.3 or 10.4 client to a Windows 2000 or Windows 2003 domain.

Any time you see a Star [*] symbol in the text, this is a link to a screenshot of what is being discussed.

  1. Setup (on the Windows DC)
  2. Open Active Directory Users and Computers (ADUC)[*] and decide on a location for the Apple Mac’s computer account. Best practice suggests creating an OU for Apple computer accounts.
  3. Create a computer account, giving it the name that you want to use for the Apple computer on your network – do not assign a GUID to make it a ‘managed’ account.
  4. Close ADUC

Setup (on the OS X client)

  1. Login with an account that has admin access to the computer.
  2. Open System Preferences and open the Sharing tab.
  3. Check that the computer’s name is the same as the one we just used in ADUC. [*]
  4. Click Show All
  5. Click Network. Select your active connection, and click on TCP/IP
  6. (If you are using DHCP then some or all of this information is possibly already delivered by your DHCP server – check with the DHCP admin)
  7. Type in a suitable IP address, subnet mask, router address.
  8. Type in a DNS server address – ENSURE that at least the first DNS server in the list corresponds to an Active Directory DNS server.
  9. Click on Search Domains, Fill in your active directory domain.
  10. Click Apply Now.[*]
  11. Click Accounts, then click Login Options, authenticating with your LOCAL admin account if asked.
  12. “Under Display Login Window As” select “Name and Password”.[*]
  13. Close System Preferences.

Testing that the client computer can “see” the network.

  1. Open the finder, and navigate to the utilities folder inside applications.[*]
  2. Open the terminal, and ping domain controllers by NETWORK NAME to ensure that DNS resolution is working properly within your domain.
  3. Ensure that name and IP are resolved correctly and that the ping actually works. [*]
  4. Stop the ping and close the terminal when done.


Binding the Mac client to the Windows Domain

  1. Run the directory access tool, which is also in the utilities folder.
  2. Tick “Active Directory”, then click Configure. [*]
  3. Fill in the Fully Qualified Domain Name of the Active Directory namespace (note, NOT the Active Directory Domain Controller!).
  4. Fill in the Computer name of the Mac. This should be the same as the one we setup earlier in ADUC on the Windows Server, and configured the mac to use in system properties. [*]
  5. Click BIND. Authenticate with your local Admin password if asked to do so.
  6. Next, fill in the details of a Windows User Account with permissions to add computers to the domain. Typically this will be an admin’s account.
  7. IF you DID NOT pre-create a computer account for the Apple Mac, then fill in the Computter OU with the details of where you’d like the account to be created, using standard LDAP notation.
  8. If you HAVE pre-created a computer account, then leave this as it is.
  9. Ensure that both tickboxes are ticked.
  10. Click OK [*]

The client will now attempt to bind to AD and join the domain.

  1. If you HAVE pre-created a computer account then you should be asked if you wish to use an existing computer account. Click OK, because that is exactly what we’re trying to do. [*]
  2. When the operation is finished, you can close Directory Access by clicking OK.
  3. At this time you can also click Advanced Options to inspect and configure custom settings. I strongly suggest leaving these alone if you don’t know what they mean or why you would want to change them. [*]
  4. Logout.

If all goes well you should now be greeted with a login window that expects you to type in your username and password [*] instead of selecting from a list.

You should now be able to log in with an Active Directory account by typing in the username and password in the traditional manner, and you can also login with a local account by specifying their username and password in the same way.


Microsoft Internet Explorer Browser: A Look Back

This was something I posted way back when browsers were the talk of the crowd. Here’s a look back on what I had to say about this browser years ago.

Microsoft has announced the next version of their web browser, which will be Internet Explorer 8. Didn’t see that one coming! Dean Hachamovitch, manager of the IE team, has some fun with ‘alternative’ names in the IE Team blog post I’ve linked to above, and asked us not to “mistake silence for inaction”.

You’ve got a point Dean, but perhaps you might try looking at it from our point of view. You know, how customers see things? You remember customers, don’t you Dean? Those people who used to use IE but now use FireFox, Opera or Safari? You know, those browsers whose developers appear to be actually doing things and haven’t kept so silent that they haven’t had to chide anyone for thinking that they’re hiding.

Gosh, when your own CEO appears to be unsure what you’re doing these days, it seems to me you ought to cut your customers a little slack. But that’s just my opinion, I’d love to hear yours. Kudos to Molly Holzschlag for asking the questions and posting the answers by the way!

Thing is though, I think Microsoft have a few problems here. First of all, I thought we were supposed to see a shorter development cycle post IE7, and that certainly doesn’t appear to be happening. A lot has happened in the browser market since IE7 was released, FireFox 3 has test versions out, Opera keeps quietly sitting in the corner getting better and better all the time and.. oh yeah… there’s this new browser appeared for the Windows platform that is based on some very interesting technologies. still, I’m sure that’s not a problem, it isn’t as if it’s supported by one of the other major IT companies… oh wait… it is.

Jeff Attwood said it very well in his recent post on IE, pointing out that given the length of time between IE6 and IE7, us Microsoft customers deserve better than that. We don’t deserve silence, we deserve some kind of proper feedback on what is going on here.

I’ve seen some suggestions around the place that Microsoft can’t innovate any more (indeed, plenty of people claim they never could in the first place). I’m not sure I agree; Microsoft can innovate as well as most of their competitors at least in some areas and at some times. XBox 360 was quite an innovative hardware design, for example.

IE4 was quite innovative in its day. Some of its ideas didn’t take off too well at the time but look at Widgets / Gadgets now and tell me that isn’t inspired by (among other things sure) IE4’s Active Desktop.

Staying with IE, the whole damn problem with that and Netscape navigator was that back in the IE3 / IE4 Netscape 4.x days was that each company innovated a little too much by trying to extend HTML in their own directions instead of sticking closely to the standards.

Of course, innovation is all well and good but all these companies are measured by the money they make, not the innovation they display. Putting a bit of meat and some salad between two bars of soap might arguably be called innovation in the field of sandwich making, but no one is going to want to eat the results, which just goes to show you that being obsessed with “innovation” at the expense of making what people want isn’t a good thing either.

So what am I hoping to see in a new IE8?

Update – Since writing this article I have found the great set of pages on IE7 interface problems at Project Cerbera, pretty much all the issues mentioned there are on my fix-list too now I’ve been reminded of them!

I think it might be time for a ground-up rewrite of the internal engine.

Yes I know that’s going to hurt, but that pain is something Microsoft will have to deal with sooner or later – the longer you keep deferring major work you know needs to be done the more the work will cost in the end, like leaving a minor dental problem until it effects your whole gumline or failing to take care of a minor problem with your car and allowing a major one to develop in its place (I must phone the garage tomorrow…).

I want to see better tools for deploying IE to multiple workstations and managing it afterwards.

The one area where IE scores a big win is on corporate desktops, where it’s installed by default and Microsoft’s ability to allow things like IE to be managed via things like Group Policies is a big win. One bad side of this corporate thing is that you often need special tools (Internet Explorer Administrative Kit or IEAK) to build a version of IE to upgrade corporate desktops. I can see where this might be useful for special deployments but why can’t I just allocate IE 7 easily via a standard MSI and configure it all in situ via GPOs? Would be nice if all parts of Microsoft could actually follow their own installer guidelines!

It should run on XP as well as Vista.

Leaving aside my opinion of Vista and the rights and wrongs of that, a lot of people out there have not upgraded yet and don’t appear to have many plans to upgrade soon. These people should not be forced to upgrade their whole OS just to upgrade the browser.

There is (as always with me) a security side to this; it seems a given that any new Microsoft app will concentrate on security (or at least Microsoft’s version of security). If there are lots of older versions of Windows out there running out of date and insecure browsers, these can easily become infected and may prove very difficult to clean up and keep clean reliably.

It needs to concentrate hard on meeting the various standards for webpage rendering.

There’s a certain amount of work that has gone into all the other browser engines out there which Microsoft’s IE has missed during it’s long hibernation after IE6. I realise it’s a lot of hard work for Microsoft to regain this lost ground, but you folks up there in Redmond only have yourselves to blame for that. And I remember the comments coming from your team around the IE7 release about how the ACID tests were not really that important and it was no biggie that you didn’t do too well on them; well even if that might be true on a practical day-to-day basis, don’t underestimate the PR problems that come from doing so badly in those tests!

The interface sucks. I’m sorry but it does. Let’s have something that looks like a Windows app next time!

Note to whoever signed off on the IE7 interface layout:I realise your primary school child is very special and talented but please don’t let them design any more user interfaces for major software releases until they’ve at least graduated high school (actually maybe I’ve found the reason for the IE8 delay?)

I know that some people like the new interface. That’s great, but please let us choose between the new layout and something more “standard”. It’s illogical, it doesn’t really fit in with your own guidelines and even on it’s own merits it has some faults (e.g. the stop button always working even when there’s nothing to ‘stop’).

When I first talked about Safari, my good friend Lewis Burgess posted a comment that I totally agreed with then and still do that one of the biggest faults with Safari for Windows is that it looks and behaves like a Mac application. Now Lewis is as keen on OS X as I am, so neither one of us was hating on Apple, but I do like my Mac applications to look and feel like Mac applications and my Windows applications to look and feel like Windows applications. Safari for Windows breaks that rule, but in Apple’s defence they can point to IE7 and say “Well Microsoft started it”. This is not a good position for either ‘side’ to be in!

Buy a damn atlas! (or attention to the little things makes a big difference)

Newsflash: America is not the whole world. I know it’s a very small thing but attention to these small details is often the very reason why some products do so well and others do not. With that in mind, is it really too much to ask you to take note of the regional settings for the rest of my computer and set the browser to use those instead of trying to ‘trick’ me into picking US English during your first run setup process? (When I see mistakes like this I wonder if it shows for people of all languages or just if you use any of the ‘non-American English’ choices)

I’m picking on this because it’s a symptom of everything that is wrong with IE7. There are a myriad of little issues that could have been resolved by checking a setting and just following what was already there, or by just picking a sensible default and making the controls to change it reasonably easy to find.


Command Lines In Your Operating System

In the cases below I’m talking about commands that can either be run natively by the operating system, or (in the case of Windows at least) are available as free addons.


Remember that Windows commands are case INsensitive, e.g. “PING” and “ping” are the same command.

pathping –

Effectively performs a tracert from your current computer to the destination computer specified in the command line, notes all the nodes found in the tracert and performs a ping on each one of them, allowing you to assess the speed of the connection you are using and spot which nodes are slowing things down. This can be considered an implementation of the Linux mtr command, which sadly isn’t listed in the section below because it isn’t natively a part of the UNIX tools in OS X.

nslookup –

Allows you to query a DNS server from your workstation. Very useful for modern TCP/IP networks where you need to see how a workstation finds things in its little world or how things in this world might find it! A classic command that has been around forever in UNIX circles, and the fact that this was such a relatively recent addition to the windows CLI underlines how much of a joke the Windows shell has been up to now when compared to… well… just about anything else really.


A very powerful command, if somewhat clunky to use, that provides access to quite a lot of the way a Windows machine manages its own user security and its relationship to the rest of a domain network.

Using variations of the NET command you can

Create and manage user accounts, add your newly user accounts to groups and set properties such as passwords (NET USER, NET LOCALGROUP, NET GROUP).
Mount network drives (NET USE)
View and manage shares on the workstation (NET SHARE)
… and many more! The NET command is often a difficult command to use once you get beyond the basics, but it really is shocking the amount of times I’ve had some obscure NET command that was written for LAN Manager years ago pull my butt out of the fire on a modern server!

netdom –

Allows you to join a domain from the command line, manage trusts between domain member workstations and domain controllers and between domains. Part of the Windows server support tools. On the one hand this is a really boring command, on the other it’s probably utterly vital to anyone building workstations for a domain-based network who needs to script the building process rather than use the automated options in the OS setup.

Note the lack of space between NET & DOM; netdom is not a subset of the NET command above.

netdiag –

Allows you to view a quick snapshot of the state of basic network settings and network configuration on a PC in a domain and is very useful for those “Can’t see the network” type of calls. Part of the Windows server support tools. Note the lack of space between NET & DIAG, this is not a subset of the NET commands above.

dcdiag –

Allows you to view a quick snapshot of the state of basic domain settings and domain configuration in a Windows Server 2000 (or higher) domain. Very useful for those “I can see the network but it takes half-an-hour to login” calls or when you suspect one of your domain controllers is in the pouts and is refusing to talk to the other domain controllers because they’ve had a falling-out over their MySpace page comments or something.

mstsc –

Arguably a bit of a silly little command to have in a list like this but I find myself frequently running it from the command line rather than from my start menu. This is partially because I always have a command line window open at work and typing “MSTSC ” is quicker than navigating to the shortcut (yes, it’s pinned to my start menu shortcut area before you ask!) and partially because I tend to use some of the more useful parameters for terminal server such as /span to match up desktop sizes or /console to grab the console window on the server rather than create a new session.

mstsc /console, grabbing the console window on a server or workstation is especially useful, in particular with some applications and desktop interfaces to server services that just don’t want to play nicely with multiple terminal sessions open on one machine.

Apple Mac / UNIX

These are all UNIX type commands based on Mac OSX 10.4.10. As always, you should double-check what you’re doing before running any of these commands on any other kind of Unix online. (Actually, I hope you’d make sure you understood what was being suggested before you followed any advice you read on my site, or pretty much anywhere else on the Internet for that matter.)

Remember that UNIX commands are case sensitive, e.g. “PING” and “ping” are not the same command.

top –

top presents a dynamic view of running processes on your system ordered by CPU use, together with a summary of the resources being consumed. This is incredibly useful if you want to see if a process is active, or (probably more likely actually) you want to see which process has decided to crash and consume all your CPU cycles and free memory, in order that you might kill (see below) it. For a non-dynamic list of running processes, consider the ps command instead.

kill –

This is how you force a process to quit in UNIX. Most often associated with forcing a hung program to quit (kill -9), kill is a signal to a process to quit which has a number of levels from forcing a quit (-9) to asking the process to stop what it’s doing and bring things to a halt as soon as it can do so in a tidy manner (-15).

Those of you who have actually been reading my past ramblings may remember that I mentioned these commands in the article I wrote after getting my first Mac. Something had hung on my new iBook and I was unaware of the Mac-specific methods of finding and force-quitting the hung process, so I actually used both the commands above to find and kill the faulty process. I think it was called “Finder” and I was quite surprised at what happened when I killed it.

rsync –

A command that is very useful for copying data from one computer to another in a very efficient way, and to keep these multiple locations synchronised so that actions that take place in one location are mirrored to the other location. Also see psync if we’re talking Mac-specific. Also see scp and rcp.

dig –

The replacement for nslookup on modern Unix (nslookup still works fine on my mac though). I tend to prefer the way dig formats its replies to queries compared to nslookup but at the end of the day this is really a matter of personal preference for most people.

One thing that might dictate your choice is if you plan to ‘pipe’ the output of your DNS query into another command, obviously you’d want to use the command that produced the most suitable output for the command you want to send the DNS information to.

alias –

A way of creating a “shortcut” to a command, including adding the default parameters that you always want to use with a command. Let’s say that you’ve heard about my BOFH style habit of tricking people into doing bad things with the ‘rm’, ‘cp’ and ‘mv’ commands to delete, copy or move files in Unix and you want to make sure that you don’t get caught out by this while playing around with a Unix or Apple tip suggested by me, you might specify an alias for the rm command that automatically turns on the interactive prompt mode (e.g. asking “are you sure” before deleting a file) with the alias command: alias rm=’rm –i’.

Many people would argue that this is a very good idea when doing just about anything at the command line if you’re logged in as root.

Remember, alias means never having to say “I’m very sorry boss but I appear to have deleted the irreplaceable files containing vital financial information.”

lsof –

Trying to find which process has locked a file? lsof is your friend, generating a list of open files by process that is holding them open. Now this is where it gets interesting; to appreciate just how powerful lsof is in unix you need to remember that just about everything on Unix is represented as a file… Including directories, open network ports and so on!

Try the command lsof –i|grep TCP to run the lsof command and to use the grep command to search the output of the lsof –i command for entries containing the line TCP to get a list of open TCP port connections on a machine, repeat with lsof –i|grep UDP to do the same for UDP connections.

Or to search for references to a folder, say /usr/bin, try lsof |grep /usr/bin – remember the grep command and the pipe character “|”, while not part of my list here these are very important parts of working with Unix at the command line.

man –

Shows the manual page for many UNIX commands. Arguably a bit of a lame end to my list of UNIX commands but then this is an important tool where the options and switches for some commands can be far too numerous to possibly remember. Remember, closing your eyes and guessing only looks cool in the movies, out here in the real world planning what you need to do then doing what you’ve planned to do is what wins the day.