Migration: WordPress + MariaDB (MySQL) + PHP + EC2 + CoudFront

I previously wrote about how and why I was migrating away from Hosteurope to Amazon AWS. The basics steps sounded very simple:

  1. Backup databases and data directories on the old server
  2. Set up the new server including all software & services needed
  3. Restore all backups to the new server
  4. Change DNS setting to apply changes to all visitors

Sounds easy, right?

Everyone who’s done that kind of thing before, knows it’s never quite that easy. Doing the first step is quite straight forward. Not a big deal. For the second step I ended up using a combination of howtos to get everything set up properly:

  1. Official AWS EC2: Linux+Apache+MySQL/MariaDB+PHP
    Tutorial: Install a LAMP Web Server on Amazon Linux 2
  2. Official AWS EC2: WordPress
    Tutorial: Hosting a WordPress Blog with Amazon Linux
  3. Working CloudFront Config for WordPress:
    Setting up WordPress behind Amazon CloudFront
  4. General WordPress on t2.nano Instance:
    How I made a tiny t2.nano EC2 instance handle thousands of monthly visitors using CloudFront

My main goal was to run the smallest instance AWS has (currently t3.nano) which costs around $3.75 per month. It turned out, MariaDB (and for that also MySQL) does not start up properly at the nano instance with just 512MB of RAM. Therefore, I had to go at minimum with the t3.micro instance.

To ensure that load spikes are being handled properly and the server will not go offline during these peak times, a content distribution network (CDN) comes in handy. The great thing about AWS is that they’ve thought of all these scenarios and of course they’ve got a CDN solution ready to deploy. It’s called CloudFront. The tricky part here is, to have CloudFront kick in at the right time, because it caches content to deliver it from its own edge locations across the globe, but at the same time WordPress generates websites dynamically. So CloudFront needs to be able to work in that environment. Setting up CloudFront properly was the part that cost most time, but it works great now.

CloudFront Network Map

I have now deployed CloudFront with multiple sites all running on the same t3.micro instance. One by one I activated for CloudFront distribution and over the past couple of days the traffic handled by CloudFront is going up continuously.

CloudFront Cache Statistics

To run massive load tests I used Apache JMeter for the first time. It’s a monster when it comes to load testing and it took me about an hour to get it running the first time. You can literally configure everything on there.

As it is when you’re setting up new things, you’ll have many “new things” you’re working with. In my case, it was the first time I used MariaDB, which is a fork from MySQL. It was also the first time I worked with php-fpm, which “is an alternative PHP FastCGI implementation with some additional features useful for sites of any size, especially busier sites”.

So far I’m quite happy with the current set up. Let’s see how this performs over time. My sites are constantly under attack from bots who are trying to figure out passwords and gain access otherwise to those sites. Yet, every site and server can and will eventually go down under just enough Denial of Service attacks. At the current attack level we’re doing OK, but let’s see how long this lasts and what adjustments I’ll have to deploy.

Please follow and like us:

Scaling a Web Server to Serve 165 Million Ad Requests per Day


I am usually not the kind of guy who likes to boast with numbers, but in this case I believe it helps to put things into perspective. For over seven years I am in the mobile app business targeting consumers directly. Over the years we have created thousands of products, most of them paid, some completely free and some with ads.

For years, I did not believe in monetization of apps through advertising. Even today, I am still very skeptical about that because you need to have A LOT of ad requests per day so that any significant income can be generated from that. At the moment, our few (around 10) mobile applications that sport ad banners, generate up to 165 million (165,000,000) ad requests per day. That makes up to 5.115 billion ad requests per month. If you compare that to a fairly large mobile advertising company like Adfonic with their “35 billion ad requests per month” (src: About Adfonic) we are doing quite alright for a small app company. We are using a mix of different ad networks depending on what performs best on which platform. Besides that we are breaking it down into country/region to use appropriate ad provider that is best for that region. Part of that also involves ad providers who do not support certain platforms natively. So built server-side components to handle such ad requests in a “proxy” kind of way that still allows us to get ads shown in apps on a certain platform that is not officially supported by the ad provider. Basically, we have a mini website that just shows an ad; and we can have hundreds of thousands of mobile devices accessing this website from all over the world at the same time.

Using cloud service providers like Amazon AWS, Rackspace, SoftLayer, Microsoft Azure, or others any can serve a virtually unlimited number of requests these day. It all depends on your credit card limit. There are obviously usage patterns of programs of applications and over the course of the day, we have ups and downs. For example, at midnight GMT most people all over the world like to use our applications and therefore they request more ads. Five hours later, we experience the lowest traffic. Depending on application store release schedules, promotions, featured listings, user notifications, external promotions like on blogs or elsewhere, unexpected sudden spikes in traffic can occur any time. “Automatic Scaling” in combination with “Load Balancing” seemed to be the magic solutions for this.

After months of running a bunch of server instances behind a load balancer we were quite happy with the performance. It was easy for us determine usage patterns and see how many servers we need to serve the average maximum number of requests without our service to fail. We didn’t think that much about cost-optimization because our cloud computing bills weren’t that high; so we didn’t really work on such auto scaling components. That had two main disadvantages: Firstly, we spent more that we needed to as we probably didn’t need half of the server that were running while we had low traffic. Secondly, we were not prepared for sudden massive spikes in traffic. With TreeCrunch on the other hand, we are building a scalable system from the ground up.

So early this week I took three hours and looked into how to implement such auto-scaling. Actually, it took me two hours to install the tools properly. However, as you can see in the above charts, the current systems works in a way that if we hit a maximum amount of average traffic per server, a new server is being created and added to our load balancer — it then immediately starts serving ads. If the average traffic falls below a certain threshold, a server is being terminated and therefore stops serving ads. To minimize server load, we run really tiny PHP scripts that are extremely optimized just for the purpose of requesting an ad the desired ad company and delivering such ad to the client (mobile app). As web server we are currently using lighttpd which is very lightweight indeed. Interestingly, we noticed that there is not really a problem with “normal” system resources to handle a lot of requests. Our CPU usage is fairly acceptable (constantly around 50%), we don’t need any hard drive space as we are just proxy’ing requests and we don’t even run out of memory. The first limitation one of our ad servers hits, is the maximum number of used sockets which is by default around 32,000 (or something like that) on Debian-based linux. That’s more or less an artificial limitation by the operating, but we didn’t play with adjusting that one yet.

Summary: In a fairly short period of time, we managed to set up a proper auto scaling policy that allows to scale up to virtually unlimited numbers of ad requests. With fairly low budgets companies nowadays can setup proper data centers, serve millions of users and maintain their infrastructure with a few very talented people and without purchasing any hardware that would be obsolete a year or two later.

I love the time I am living in. Every single day something new and exciting pops up.

Please follow and like us: