- Beginning in the late afternoon of Monday, July 21, I received emails
from a number of HASA site visitors that something was terribly wrong
with the site. Because of regulations at my workplace, I had to wait
until I got home to view the site. My heart sank when I saw the errors
affecting almost every page of the site. It was obviously malicious
work, so I shut off the site to prevent further access.
Looking at source code, I saw information that should not have been
quick Google search revealed the problem. There had been a worldwide SQL
injection attack on ColdFusion sites that ran on IIS web servers. Many
sites had been brought down and a lot of data had been corrupted.
Luckily, the attacker had only tried to insert code to convert site
links to go to his own web site, and had not tried to delete data or
erase databases. The ColdFusion community had responded quickly. By the
time I was looking for information, the exact attack had been
identified, various clean up scripts had been posted, and I had the
tools necessary to fix the database.
What is a SQL injection attack? It is the nastiest, most destructive web
site attack out there. In general terms, a coder takes a site form or
URL, looks for a variable that appears to be an ID used by a database to
locate a record, such as a story. Most database driven web sites work
like this. The coder then resends the URL or form with a string of
database commands (Structured Query Language statements) and tries to
take over your database. If their SQL string is written well enough and
if the database under attack is not protected, these commands can do
anything from insert text to delete the entire database to (under very
rare conditions) take over the server itself.
ColdFusion has a number of features to foil SQL injections right on the
server and there are coding practices to help prevent the attacks. Just
over a year ago, I had spent a month retrofitting much of the site to
protect the main data tables and search functions from attacks. However,
real life demanded my attention, and I did not get all the tables
protected. Some of the lookup tables, like Story Era, were still
vulnerable, and that's where HASA was attacked. In addition, the SQL
statement used for the attacks was very sophisticated and bypassed most
of the server-level protections.
The first thing to do was assess the damage. I ran some scripts and
discovered there were 29 fields in 11 tables where trash data had been
inserted. None of them affected the main story, forum, comment or review
tables, which was a great relief. Also, non-HASA databases had not been
affected, which meant damage was contained. Next, I ran a scrubber on
those tables which located the bogus text and stripped it out. The
biggest advantage here was that I did not have to run a restore from a
backup, which would have lost content added to the site since the time
the backup was run.
Data restored, I now had to track down those vulnerability points in the
site. That is what has taken so long as every individual page in the
site that an external (not logged in) user can reach had to be examined
for all database connections and then those connections had to be
evaluated for whether they were injection points. After the pages were
reviewed, corrections made, and new code posted to a hidden site for the
testers to check for bugs, all the forms had to be gone over. Forms are
more dangerous than URLs, though more difficult to compromise, because
they are the most common way to insert data into a database. The forms
were completed yesterday. After that, the revised code was merged with
the production code, uploaded again to another protected test area and
put through its paces. That went on until pretty late last night.
There are a few glitches left. One form, the Knowledge Base search in
the Research Library, could not be fixed and has been taken offline. The
index search and all of the individual section lookups the Research
Library are working. Also, the pop-up forum window attached to story
pages will throw an error message if you try to use it. The Who's Online
feature for logged in members does not appear to be working in
production. There are a few conditions on a few forms we were not able
to reproduce, so it's not 100% certain that they are working.
More testing will continue over the next month to be certain all access
points were caught and to fix any bug accidently introduced during the
fixes. If you encounter an error message or you see something that just
seems wrong on the site, please use the Contact form and send me an
email at Tech Support.
This was the worst technical problem the site has experienced in its
seven year history because of the data corruption. We got away with
correctable damage, but it could have been much worse had the attacker
been trying to destroy data. My deep thanks and appreciation for the
site members who alerted me to the attack and provided their assistance
to get the site fixed.