Cost effective development environments on AWS

I believe many companies struggle when it comes to hosting development environments for their clients. At least if they have to host all of their clients development environments on their own hosting without asking clients money for that.

The problem we encountered

At Magebit we do different kinds of eCommerce services. Mostly on Magento that require decent hosting to run the sites smoothly. We offer support with no minimum monthly payments so we end up having clients that do not pay for multiple months and we need to still have their development environments working.

Let’s put everything on a single instance

This was the first idea we had — not that good one though. The first issues started when we had to import data for full site build development environments. The more clients we get, the more issues we have as we needed to scale up the instance to keep the sites running smoothly. We wanted to put the instance on a schedule so it turns off at nights to save money but couldn’t do that as when it is night at our office, some of our clients have just opened their eyes and start their workday.

Sure, the costs were smaller but the quality we provided on development environments was not the one we intend to serve our customers. It can be an okay thing to do while freelancing and working with small sites but not at our level for sure.

Game changing plan

This might be old for somebody out there but for us this idea was something new and exciting.

We created a plan where we have separate instances for all our development environments. Some may think that cost will be crazy at this moment but actually the costs aren’t high at all if they are turned on on-demand.

Automating it all

Sure, this can be done manually to keep the costs low but as the instances and environments grow the pain managing them all would grow much faster than we could think of (at least I think so).

We automated it all. From easy environment setup to automatic instance control. Instances are created by our DevOps and then they just run a single command and magic happens:

1. Environment gets prepared for certain site hosting (Magento, Wordpress, Laravel or anything else). I know many use Chef for this but because Ansible is easier and fast to learn we chose Ansible over Chef.
2. CloudWatch event gets created or updated to monitor certain instance state.
3. DNS records are created or updated in Route 53 with the appropriate DNS Failover rules to direct traffic to Master Instance in case the EC2 Instance is sleeping.
4. Master Instance is configured to determine the domain the visit is coming from, locate the associated instance and turn it on via AWS SDK. While it is turning on it shows a stand by page that tells the visitor that the instance is turning on and that he will be redirected to the working site once it is on.
5. Automatic check script whether the instance is being used or not is executed via cron. We check for active or recent shell activity as well as webserver activity. During business hours the script checks activity for 4 hours and on weekends and outside business hours it checks whether there was an activity during the last hour. If there is no activity the instance gets shut down.

Looks cool but that is not all.

Keeping an Elastic IP for all the development environments can also increase the expenses pretty much when it comes to providing support services for variety of clients.

That is why we chose not to use Elastic IP and add little more automation to the instances. On Public IP change the EC2 instance sends info to Route 53 and CloudWatch to update the records. DNS record to update to the new IP and CloudWatch to monitor the correct (new) IP address. This is also done via AWS SDK.

Results

Let’s assume we are running 100 development environments hosted on Linux EC2 t2.micro instances in EU Ireland region. Assuming all of the instances are used 40 hours per week as some of them are used more and some of them less (majority less so the cost with on-demand environments actually will be less).

Approximate monthly prices with and without the above implemented (only for EC2 usage):

Regular environments: USD 1025On-Demand environments: USD 278Total savings per year: USD 8.9k+ (~73%)

Impressive, huh?If you are interested in hearing more or need help accomplishing something similar for yourself — contact us and we will be more than happy to help. We are open for any kind of AWS challenges.