On Tuesday, 14th of January 2014, our main application server got hacked by a botnet, which got root access, using SSH to our server. Since the attack wasn’t done by a human and since it wasn’t targeted, we strongly believe that no data was stolen and no data got corrupted. Despite that we took the most strict security measures, we could, to ensure our users’ privacy.

the incident

The security breach of our application server was caused by three main reasons:

1. we hadn’t disabled ssh login for root
2. we hadn’t disabled ssh login with password
3. we trusted our provider’s 12-character long random password for root

The intruder was trying to get access as root at about 04:00 GMT, while it managed to gain access at about 09:00 GMT. After gaining access, the botnet touched some of the system binaries (like ls) and deleted private-key files. The lack of monitoring on our auth system, combined with the lack of notifying us on these occasions led us on realizing the incident, after the intruder gained access.

what did we do

When we realized about the intrusion, we immediately took our servers down, completely. Therefore we had to ensure that

1. no data had been stolen or destroyed
2. no malicious software should be on our server
3. privacy of our users would not be compromised
4. this would not happen again

First, we disabled ssh login for root and login with password for all of our servers. Then, we created a new application server and a new database server and we moved all of our data to these servers and shut down the old server, completely.

Next, we wiped out all the sensitive data that we had. That means that we replaced all of our users’ private/public key-pairs with new ones, we deleted all sourceLair public keys, of our users that had authenticated with GitHub, from GitHub, we revoked all GitHub access tokens and reseted our OAuth Client Secret. Last, we reseted all of our users passwords and replaced our obsolete non-salted MD5 encryption with salted SHA-256 encryption with random 32-character long salts for all passwords.

We didn’t take the least risk of this happening again, so afterwards we replaced all of our server passwords with computer-generated 20 character long random tokens. Additionally we used iptables to restrict access to our database server only from our application server, from within the local network.

what did we learn

The most important thing we learned is that security of our service is and should always be the first priority for us. The truth is that in favor of developing rapidly new features and fixing bugs, we underestimated the potential intruders out there and left some vulnerabilities open, by postponing their solution for a later time-period, because of believing that this would never happen. From now on the security of our service is and will always be our top-priority, since the privacy of our users’ data is the thing we care about most.

The next really important thing that we learned from this incident is that we trust no one that will not try to break into our system. That means that we will take always, all measures available so as not to take the least risk for another breach, in any way possible.

what happens next

We already took all the necessary measures so as to make sure that this won’t happen again.

We continuously roll out new security measures that result in enhanced multifaceted protection of our users personal data. The goal is to run all of user-related actions into isolated containers, within the next month. We have already done great progress on that, since most user-related actions are isolated.

Last, our users have to make sure that they will delete the old sourceLair public key from any server or service that has been put on. Additionally, within the next 24 hours, all of our users, will receive an email informing them about this incident and suggesting them to update their passwords at sourceLair.

We really care about our users’ privacy. As we said before, it is our top-priority. We are really sorry that this happened and we strongly apologize to all of our users for that. We will make sure that such incident won’t happen again.

Here you can find a slightly extended version of this report.