An Attacker Takes Over a Cloud Server

Hacked in the Cloud

Amazon's Elastic Compute Cloud or EC2 provides an easy and economical way to run servers "out in the cloud". In just a minute or two of clicking through some web dashboard screens (or running their command-line interface, or making API calls) you can have one or more servers running in their facilities in the eastern US, California, Oregon, São Paulo, Dublin, Frankfurt, Tokyo, Singapore, or Sydney. The price can be as low as US$ 0.02 per hour.

It's easy, it's cheap, but it isn't necessarily safe.

Nor is it especially dangerous. In fact, Alert Logic's monitoring service shows that in-house computing environments have significantly more problems than those hosted either in the cloud or at service providers.

But wherever your server is, you must be careful.

Amazon
ASIN: 1449319262

Amazon
ASIN: 1118571835

Background

Amazon provides thousands of images of systems. These are templates that you can have copied onto a virtualized platform and started. Your SSH public key is copied into place on the server as ~/.ssh/authorized_keys in the home directory of either root or the ec2-user account. Amazon calls these ready-to-run templates Amazon Machine Images or AMIs.

An image or AMI is the standard image waiting to be copied and started as a running instance.

Bitnami is an open-source project that creates installers for web application and development stacks. For example: an Apache web server front end, plus a MySQL database on the back end, plus some programming in PHP or Python to tie them together, plus some automatic monitoring and backup. Bitnami Cloud Hosting provides Bitnami-based AMIs in Amazon's EC2.

PHP is a programming language that runs on a web server where it can dynamically generate components of the page. For example, I have Google AdSense ads on my pages. To cause your browser to ask Google's ad server for the ad content in each of the three ads you see above, the page needs to have a block of HTML code defining a floated or centered division, and inside of that, several lines of JavaScript defining the ad itself. But instead of copying a multi-line block into place I simply include one line for an ad:

<?php include($_SERVER['DOCUMENT_ROOT'].'/ads/responsive-infosec.html'); ?>

When your browser asks for this page, the server first inserts the referenced file into place.

You can do much more with PHP, and among many other tasks, Bitnami images use it to provide a web-based interface to a MySQL database.

The Vulnerability

Some of these Bitnami AMIs contained a vulnerability in the PHP language. The vulnerability is described here, it was a problem in PHP version 5.3.* before 5.3.12 and in 5.4.* before 5.4.2.

The Exploit

The first thing we noticed was this e-mail from Amazon telling us that one of the Bitnami instance we were running had launched a denial-of-service attack against a host in Sweden (the target IP address 188.126.73.62 was allocated to portlane.com in Stockholm):


		From: Amazon EC2 Abuse [mailto:ec2-abuse@amazon.com]

		Sent: Thursday, October 31, 2013 12:06 AM

		Subject: Your Amazon EC2 Abuse Report [14200733512-1]

		

		Dear Amazon EC2 Customer, 

		

		We've received a report that your instance(s): 

		

		Instance Id: i-90b0fcf4

		IP Address: 50.19.76.115

		

		has been making Denial of Service attacks against remote hosts
		on the Internet; check the information provided below by the
		abuse reporter.  

		

		This is specifically forbidden in our User Agreement:

		http://aws.amazon.com/agreement/ 

		

		Please immediately restrict the flow of traffic from your
		instances(s) to cease disruption to other networks and
		reply this email to send your reply of action to the
		original abuse reporter. This will activate a flag in
		our ticketing system, letting us know that you have
		acknowledged receipt of this email.  

		

		It's possible that your environment has been compromised by
		an external attacker. It remains your responsibility to
		ensure that your instances and all applications are
		secured.
		
			This link
		provides some suggestions for securing your instances.  

		

		Case number: 14200733512-1

		Additional abuse report information provided by original abuse reporter:

		* Destination IPs:

		* Destination Ports:

		* Destination URLs:

		* Abuse Time: Wed Oct 30 22:10:00 UTC 2013

		* Log Extract:

		<<<

		We received a report from a confidential third party source
		of Denial of Service (DoS) attacks launched from the
		implicated instance listed above towards 188.126.73.62.
		


		Please investigate the instance in question and respond
		detailing the corrective measures you'll be taking to
		address this activity.
		


		In the mean time, we have blocked all ports except
		incoming TCP port 22 and 3389 from the instance to
		prevent further abuse.
		


		If you believe that you were compromised by an external
		attacker, the best recourse is to back up your data,
		migrate your applications to a new instance, and terminate
		the old one. Attempting to repair a compromised instance
		does not guarantee a successful cleanup in most cases.
		We recommend reviewing the following resources to ensure
		your EC2 environment is properly secured: 

		

		Tips for Securing Your EC2 Instance:

		http://aws.amazon.com/articles/Amazon-EC2/1233 

		

		
			Security Best Practices

		

		AWS Security Center:

		http://aws.amazon.com/security

This instance had been started around 17:00 UTC on Monday. In less than two and half days an attacker had managed to find it, break in, and start using it as a largely untraceable tool to attack others. The cited "Abuse Time" was 22:10 UTC on Wednesday. Amazon's message went out at 00:06 PDT or 07:06 UTC Thursday.

Once the attackers know about this PHP vulnerability, they follow a fairly simple path. First, figure out which AMIs contain the vulnerability. The AMI name could announce the presence, something like bitnami-with-php-5.3.9 or similar. Or maybe someone needs to do a little low-cost experimenting to see which AMIs contain it.

Second, devise an automated way of scanning the Amazon Web Services blocks of IP addresses looking for running instances containing this vulnerability.

Third, start breaking in and abusing those resources.

Cleaning Up

The very simple fix for this specific instance was simply to terminate it. The instance had already served its purpose.

That might seem a little unusual or drastic, but a lot of times a cloud instance is only needed for a short time. Why pay to run it when you don't really need it? Many cloud servers are ephemeral, running for a very limited time. Of course, this makes it even harder to notice vulnerabilities! At least it's also more difficult for the bad guys...

Nine days later, news came out from Amazon:


			From: Amazon Web Services, Inc.

			Sent: Saturday, November 09, 2013 12:12 AM

			Subject: An Important Message from AWS Security

		

		Dear AWS Customer,

		

		Your security is important to us.  Bitrock, the creator of
		the Bitnami AMIs published in the EC2 Public AMI catalog,
		has made us aware of a security issue in several of their AMIs.
		EC2 instances launched from these AMIs are at increased risk
		of access by unauthorized parties.  Specifically, AMIs
		containing PHP versions 5.3.x before 5.3.12 and 5.4.x before
		5.4.2 are vulnerable and susceptible to attacks via remote
		code execution.   It appears you are running instances
		launched from some of the affected AMIs so we are making
		you aware of this security issue. This email will help you
		quickly and easily address this issue.

		

		This security issue is described in detail at the following
		link, including information on how to correct the issue,
		how to detect signs of unauthorized access to an instance,
		and how to remove some types of malicious code:

		

		link

		

		Instance IDs associated with your account that were launched
		with the affected AMIs include:

		

		i-92d887f4        us-east-1

		

		i-e9ea7a91        us-east-1

		

		i-ebea7a93        us-east-1

		

		i-edea7a95        us-east-1

		

		i-efea7a97        us-east-1

		

		Bitrock has provided updated AMIs to address this security
		issue which you can use to launch new EC2 instances.  These
		updated AMIs can be found at the following link:

		link 

		

		If you do not wish to continue using the affected instances
		you can terminate them and launch new instances with the
		updated AMIs.

		

		Note that Bitnami has removed the insecure AMIs and you
		will no longer be able to launch them, so you must update
		any CloudFormation templates or Autoscaling groups that
		refer to the older insecure AMIs to use the updated AMIs
		instead.  

		

		If you wish to continue using the affected instances you
		should execute the following command on each instance to
		prevent unauthorized access: 

		

		sudo rm -f /opt/bitnami/apache2/cgi-bin/php-cgi /opt/bitnami/apache2/cgi-bin/php-cgi.bin 

		

		You can learn more about how to access your instance via SSH on the Bitnami wiki: 

		link
		

		

		In addition, there are other actions you can take that will further increase the security of your instances: 

		

		1.  Restrict inbound access to your instance to a specific IP range. This is especially important for administrative ports like TCP 22 and 3389:
		

		link
		

		

		2.  If you are using password-based authentication, be sure to rotate any existing passwords.
		

		

		Additional assistance and documentation related to AWS security best practices can be found at:
		

		link
		

		

		We hope you find this information useful.  Please don't
		hesitate to follow up with us if you have any questions
		or concerns.  

		

		Regards, 

		

		AWS Security

I like a number of things about this.

Let's look at the initial report first. Before Amazon had found the underlying cause, they pointed out that our instance was hacked without acting like the Internet Police. They still use the word "please" multiple times in this message, good for them!

They also made some changes, modifying the firewall rules to block all but SSH and RDP, the safe administrative control mechanisms for Linux and Windows, respectively.

They close by suggesting security references at various levels of detail.

The second message is even better. Now Bitrock has told Amazon what is going on. Amazon has told us how to find our instances still at risk from this vulnerability. They then explain how to work around the problem on an instance we want to keep running, and (more usefully) how to migrate an operation to a compatible instance based on a patched image.

Bitrock has removed the vulnerable images and told us how to find compatible replacements.

With Amazon's warning about CloudFormation and Autoscaling, we should be able to avoid an accidental denial of service inflicted by forgetting to modify our cloud architecture design.

Finally, we are given further suggestions on how to get guidance from both Bitnami and Amazon.

I think it's important to notice that the AMI source and the cloud provider communicated with us, spotting a problem and providing guidance, but they did not step in and clean up the mess for us.

Some people will argue that a cloud provider should do all that and more for us, but they are missing the point. The standard definition of "cloud computing" requires on-demand self-service, otherwise we won't have the rapid elasticity and measured service at very low cost.

Yes, you can get remotely hosted managed services run by a third party. It will cost a lot more, be far less flexible, and it won't really be "cloud computing" by the commonly accepted definition.

Back to the Security Page