Why we moved from Hong Kong to Germany

From January 2005 to December 2007 I lived and worked in Kunming (PRC) and from January 2008 to March 2018 I lived, worked, got married and got a beautiful child in Hong Kong SAR. Moving back to Germany after living in Asia for 13 years was quite a move. Somehow I feel like having to write down why we made that move in the end.

Summerfest at Kindergarten

Education – The Main Reason

It’s a common theme in Hong Kong: When folks find out they are to become parents they start freaking out – big time. That’s largely due to the clusterfudged situation of the Hong Kong education system. I’m sorry, but there is really no better way to describe that.

I remember when my wife was 5 months pregnant we had a nice dinner with friends – a couple who were roughly at the same stage. During that dinner he dad-to-be asked me: “Which school is your child gonna go to?” I didn’t know how to respond. Come to think of it, we didn’t even know the gender of our baby yet!

Parents are freaking out

To be honest, it really is like that. Even before you’ve given birth you should know where your kid goes to school. If you want it to be a good school, you got to make sure the child goes to the right kindergarten. To get into that right kindergarten, the child needs to go to the right nursery. Of course, there are entry tests (think job interview) for 2 year olds. So basically you’re building up your child’s resume from the birth onward. It’s like an arms race!

Private is expensive

All of this is regardless of how ridiculously expensive international schools are in Hong Kong. Let’s look at the German Suisse International School for example. The tuition range is HK$148,960 – HK$194,100 (€16,286 – €21,222), which can be paid at HK$19,410 (€2,122) per month, but before you are even allowed to join the school and pay tuition, you need to pay a debenture of about HK$500,000 (€54,668). Source GSIS Fees.

Let’s assume you have that money and are willing to pay for it. You still need to get (your child) in by going through interviews, submitting resumes, etc. at kindergarten level. It might happen that your 5 year old gets three photos placed in front of her: one of President Xi Jingping, President Donald Trump and President Vladimir Putin. Then she gets asked to tell their names and their current title. (I’ll let that settle…)

Public is in Chinese

Of course, there is the public school system too. However, most of such schools teach in Cantonese or Mandarin. For us to be able to support our child during education – may it be just simply being able to understand what the homework is about – we would need an English-medium school. There are still a few such public schools in Hong Kong, but as you can imagine, these are quite popular and therefore quite hard to get in as well. Their school fees are affordable as well at about HK$3,000 (€330) per month or so.

Memorize or Learn

Besides the “getting into the school” and “paying for school” topics, there is – most importantly – the whole topic of how your child is being educated. The style of teaching and what is best for your child.

Still today, many styles of teaching all across Asia are focused on memorizing knowledge rather than learning, understanding and creatively using your free mind to come up with a solution. It is something I personally embrace and it is also very important to my wife who was fortunate enough to receive a liberal arts education in the US. We want our children to be free thinkers who can create new things and make the world a better place because they can think way out-of-the-box. Think of it like that: If high ranking government officials in Hong Kong and China send their own kids out of the country to boarding schools and colleges in the UK, USA, Canada, Australia — you know what they think of their own education system.

Roaming BL City Arboretum

Health Reasons

In Germany the air quality is definitely better than in Hong Kong. The kind of food you can buy (at affordable prices) is also of higher quality here than what you can get in Hong Kong. Though the differences are (in my personal opinion) marginal, they are definitely there.

Future of Hong Kong

1997 Hong Kong was returned to China, which was sealed by a joint declaration signed between the UK and China, which guarantees Hong Kong’s extensive autonomy until 2047. Almost half of these 50 years have passed now. Hong Kong has prospered like never before, but politically, nobody knows where that incredible city is headed towards. No one has an answer about how Hong Kong will look like after 2047. The only thing Hong Kong locals are experiencing is that the China mainland is slowly but steadily creeping more and more into the everyday lives of Hong Kong people. There are many small cases that happened over the 10 years I have been there and it really felt it is getting more and more. Maybe that was objectively not the case, but let me give some examples.

The Chinafication Hong Kong

Things like this happened: Booksellers who sell books critical to the China mainland political system disappear from Hong Kong and suddenly pop up somewhere in China at a police station. China law is being practiced in Hong Kong at certain border crossing stations due to ludicrous reasons, simply to create precedence cases of that. Independent, reputable and China-criticizing media companies that get bought over by pro-party mainland companies. Mainland liaison officials constantly targeting demonstrators with public statements that get broadcast all across HK, which in turn tries to influence public opinion and split the society. Liberal lawmakers getting disqualified for office (though some of them were serving as lawmakers for over a decade already) over an oath-taking ceremony that they felt being too pro-China. China having the power to re-interpret the Basic Law (kinda mini-constitution) that Hong Kong exists by. More and more public schools teach in Mandarin rather than Cantonese or English. Mainland Chinese business folks buying up property in Hong Kong en-masse in cash – probably with some kind of dodgy money – driving up prices and making it harder and harder for younger generations to find places to live. And so on …

If you look at each of these incidents individually, they are not that important. Yet, combining them, looking at the bigger picture and the (most likely) goal of the final reunification of Hong Kong with the communist mainland China; you can’t shake the thought of slow assimilation of Hong Kong into the China-System. I might be paranoid here, but it definitely has that kind of feel.

My wife and I were “foreigners” in Hong Kong. We possess permanent residency there, which means we can go back any time we want and don’t need a work visa. We are flexible and if SHTF it would have been easy for us to simply get out. But we also know a good number of locals who don’t have that kind of flexibility.

So again, this wasn’t a major concern for us at the moment, but it was a topic of concern and I truly hope Hong Kong’s people and politicians find a common ground with their China Mainland counterparts for a prosperous future of Hong Kong that guarantees individual freedom, rule of law and an open society as Hong Kong citizens enjoy right now.

Space

Another reason for us to move out of Hong Kong was space. As I mentioned earlier, Hong Kong’s property prices are ridiculously expensive. We were fortunate enough to live in a 990 sqft (91 sqm) apartment that we owned there. Rent for that would have probably been in €3,000 per month range or so. Yet, we only had 1 bedroom for the 3 of us. It was tight, but we managed for quite a while.

Besides normal living space, there are also these moments I remember from my childhood in Germany. I was able to just open the door, go outside, ride the bike around with friends, play somewhere in the woods and so on. That kind of play is simply not possible in Hong Kong. Because everything is so dense, there are people, cars, buses, trains everywhere and up until a certain age, you basically can not let your child walk around alone in HK.

Park with ducks in the city of Eschwege

Work & Friends

These were probably the only factors that were counter-arguments for moving out of Hong Kong. The majority of our friends are in Hong Kong — though quite a number have left already to other countries as well. Our work was in Hong Kong and everything we’ve built evolved around that incredible city. Though, most of work can be done online nowadays, certain things we’d like to do in person. I personally liked working in Hong Kong, I liked my colleagues and I liked the kind of work we were doing. It was great, inspiring, challenging and yes, even the controversial conversations I’ve enjoyed. So again, work and friends were the arguments for us staying in Hong Kong, but everything else eventually outweighed those.

Family

When it was just my wife and me, it was easy to live in Hong Kong. Even without any immediate family of either of us in HK, we could live there comfortably. We had the freedom to move around, go on holidays and visit our family whenever we felt like it.

With a child your life suddenly becomes quite a different one. You want her to grow up with her grandparents. That’s great for the child, but also for the grandparents. Of course, it doesn’t hurt to have a baby sitter option at hand, but that wasn’t that much of a concern for us. At the same time, for your child to be with her cousins and other relatives is quite something. It’s just fun.

Conclusion

While there were many reasons for us to move back to Germany, we truly loved Hong Kong and it was not an easy decision to move to Germany. Hong Kong is a very diverse city and in the 10 years I was living there, it never stopped to amaze me. There were so many hidden gems that surprised us and that happened again and again. Yet, the final reason for us moving was really for the sake of our daughter. She’ll get some great education, she’s got space to grow, play and roam around, she’ll grow up with at least one side of her family and she’ll be among many kids of her age whom might or have already become her friends.

We’ve moved to the small town of Bad Langensalza where I grew up in. Some of my friends still live here, most part of my family lives here too. It is a very different way of live here. It’s much slower (especially the Internet connection!), but it is great in its very own way.

We’re now in the process of building a house for us to live in. Again, it’s a much slower process than I anticipated, but I am sure I will share some stories about here as well. You’ll laugh your butt off! Stay tuned. Subscribe to my blog or join my Facebook page.

P.S.: Check out www.weitsprungmeeting.com (long jump meeting) which took place in Bad Langensalza two weeks ago. The world’s best long jump of the year took place right here in the green heart of Germany. Can you believe that?

A fresh look

So yesterday and today I’ve been tinkering with my blog and gave it a quick new look. The old style was from 2005 and I figured a 13 year old look might not up to our trained eyes yet. The new look still has some quirks like the cut off images on the right, but it is already way better than before. When I have time, I’ll probably fix those.

Just for the sake of documenting my progress, let’s check out some comparison screen shots.

kozen.de look yesterday:

kozen.de look today:

 

It’s a bit easier on the eye now. What do you think?

Why do I blog (again)?

That’s an interesting question and indeed the kind of the first one I was asked when I relaunched my blog yesterday.

I am using various social platforms for years now: Facebook, Twitter, Instagram, SnapChat, Youtube, Foursquare, WeChat, LinkedIn, Xing, BBM, Reddit, you-name-it. They all have the certain purpose — though, there are some that I really don’t quite get (Foursquare and Snapchat for example). Anyhow, none of them fits all my needs.

I’d like to share content in various formats. I like written content most, often accompanied by photos or maybe video, but sometimes it’s really just text. With other posts, there might not be any text. I want all my content to be accessible on the world wide web by everyone. So exclusively publishing it inside a walled garden like Facebook, is not that good IMHO. Audio is also a media I’d like to get into some day — though I still feel really awkward hearing my own voice.

Some content may be of more personal nature, some content maybe of more business / work-related nature. So context-related platforms like LinkedIn and Xing are not working all the time either.

As you can see, it is not that easy to choose a main platform to publish content on — if the requirements are as loosely defined as I defined them.

Furthermore, by posting my content on my own blog I retain full control over it. That’s not that much of a concern to me, but it’s a nice thing to have. I am happy to share my content and I truly hope it may help people in their life. One way or another. I have benefited in various ways from a lot of other people who shared their insight – may it be personal, business, tech or otherwise related. Perhaps someone might find some of my new content useful in future.

One of my greatest stories is that of Juergen who commented on my blog post Hong Kong ID Card – Fingerabdruck und Biometrische Gesichtsdaten inklusive, which I wrote 10 years ago in July 2008. A few years later, we actually met and became friends. He even ended up renting a room in our office in Hong Kong when we were renting out “serviced offices” there. Only then I realized it was actually him who commented on my blog post – way before he actually came to Hong Kong. That’s the kind of online content that brings people together. That’s what I want to share and how I want to build relationships with my readers with whom I then hopefully become friends as well.

Change is Good – Life in Germany Launch

It’s been almost 4 years since my last article and 13.5 years since my very first article here on kozen.de. Things have changed, priorities have shifted and life just keeps changing.

To celebrate and document that change, I’d like to revive kozen.de and publish a few more updates here for my friends, family, colleagues, acquaintances and everyone else who is looking for what I have to say. As usual, it might not always make sense to everyone, I might not be very logical (though, I try to be) and some topics might be boring to some of you. Yet, I believe, I have some interesting stories to tell about our new life (back) here in Germany now and everything that’s happing around that and around the world, where ever I am involved or interested in.

In the following weeks, you can expect to see:

  • Visual relaunch of this blog
  • Stories about building a house in Germany
  • How it is like to live in Germany
  • Thoughts and articles about cryptocurrencies, startups & ventures, property, tech and projects I’m interested in

If you want to stay up-to-date, you should subscribe to this blog or like my page on Facebook.

Yo compared to our Battery Watch app – Why Investors Value Yo At $10 Million – and – if Yo = $10m then Battery Watch = $20+m

I’m still baffled by the bold move of Silicon Valley investors valuing the company behind the Yo app at $10,000,000 USD. According to themselves Yo is:

The simplest & most efficient communication tool in the world.
Yo is a single-tap zero-character communication tool.
Yo is everything and anything, it all depends on you, the recipient and the time of the Yo.

To an ordinary mind like mine the app does not sound very interesting and from a targeting point of view too general than you could actually define a specific target audience for it. Yet, I was intrigued by the simplicity and most importantly by the buzz around it. About a month or so ago, I decided to download Yo. Right after it launched publicly, actually. I have 6 contacts on there and in the beginning it was kind of fun to Yo people just for the heck of it. Nowadays I only Yo ‘back’ if someone Yos me first.

The Wall Street Journal apparently has made a very compelling case that Yo might become a communications and marketing platform to allow people and brands to communicate with each other in a more simplified yet sophisticated way. For example, adding a link (that opens a website) to a Yo notification. To be honest, the real key word here is platform. That is what gets investors all nervous to throw money at you. To them, there is no benefit in funding just another app. It’s the platform behind the app that makes all the difference. Based on the anticipation of creating a simple and very direct marketing platform to bring brands closer to consumers makes check books open magically and throw money at an app that actually is absurd. In fact, Michael Santoli of Yahoo! Finance said: “When almost everyone immediately agrees that something is absurd – how can it possibly fail?”

So let’s look at the plain numbers thus far. According to the Yahoo! Finance article, Yo has “been downloaded 1 million times and has around 50,000 active users.” Based on a per active user valuation (which is how many tech and especially social media companies are valued by) that makes a staggering $200 per active user valuation. Compared to the $19bn WhatsApp takeover by Facebook which valued each of their 450 million active users at $42 per such user. The Yo valuation at 5 fold seems a little high. Anyway, another interesting aspect to look at, is the “stickiness” of an app. We at S4BB Limited look at a metric called “Retention” to determine the stickiness. In very simple terms, that’s the ratio between total downloads and active users. So at 50k active users and 1m total downloads, Yo has a retention of 5%. Common industry standard for a good / sticky app is something around the 20-25% retention mark.

Just to compare this to our recently re-launched Battery Watch app on Android: 193k total users, 103k active users, 58% retention. Here is a screen shot taken this morning in our Flurry app analytics console:

Based on the number of active users our valuation for Battery Watch should be twice the Yo valuation so we’d end up at $20 million. Based on the retention rate our Battery Watch should be valued at 11.6 times the Yo valuation. That would be an outrageous $116 million. Plus, compared to Yo, Battery Watch is even profitable already. Yo, b***!

Sure, Yo has all the publicity, the advantage of being located in Silicon Valley and the platform (hope) story behind it. While we don’t have the first and second point, our platform is currently being designed as well and will be launched in select markets very soon.

It remains to be seen how successful Yo can be in the long run. Right now it seems to be hip – at least on the investor and media side – but to build a long-term sustainable business model on top of that will be challenging. I wish the Yo team all the best in their endeavors.


More about Battery Watch: Battery Watch is a very simple, useful and yet fun battery monitoring tool. It combines essential battery information, useful TSA / UK DfT-compliant battery warnings with funny voices that make such a usually “boring / ordinary” utility app really fun to use. Try it for free through Google Play: http://bit.ly/BWatchA or BlackBerry World: http://bit.ly/GetBatteryWatch

About the author: Patrick is co-founder and CEO of S4BB Limited. An independent, boot-strapped mobile app company that has published over 48,000 apps for Android, BlackBerry and iOS. He is based in S4BB’s global headquarter in Hong Kong. Patrick has also co-founded and invested into other startups like Sky Drone, Toshi, TreeCrunch Limited, Skylab Mobilesystems Limited and Slate Takes Limited. You can find his full bio at www.patrick-kosiol.de or connect with him on LinkedIn, Xing or AngelList.

Running A Startup Isn’t Easy, Working At One Isn’t Either


It has been 9.5 years since I left Germany for China to help start Skylab Mobilesystems Limited. One year after I co-founded S4BB Limited and subsequently other ventures like TreeCrunch Limited, Slate Takes Limited, Sky Drone and most recently Toshi. Since moving from Kunming (Yunnan, China) to Hong Kong I have been involved in the local startup community by as much as I can just make time for it.

Last week for example I was filling in at Web Wednesday as it seems there were not enough mobile app developers who had signed up for the 6th edition of their famous Battle of the Apps. That night was a great experience with a rather small, yet interesting crowd I was happy to talk to before and after my stage appearance.

Some younger folks I talked to are thinking about starting their own project – I believe they were recent university graduates. Having the courage to launch your company in Hong Kong – especially when you are a Hong Kong-local – is a huge thing in my eyes. While in the west (and with “west” I mainly mean North America) the startup culture is being is embraced, it’s somewhat glorious and something to strive to. Here in Asia it is widely being avoided by young folks, though. There is a lot of tradition in play and a lot more of of parental control. As opposed to Europe, in Asia parents still have a lot to say when their fresh uni graduate kids look for their first job. Often the parents have paid hefty tuition for their kid’s education and therefore, they demand them to get a “proper” job at a large firm like a bank or an international conglomerate. That doesn’t event stop at folks who want to start a business themselves, it is going as far as getting the right job right after uni. In fact, I heard from some startups who did actually create a “care package” for a job applicant’s parents to show them how it is like to work at a startup.

When we started off initially, we did not take any external funding nor did we have any ‘family support’. In 2005 we just thought there might be a market for software running on mobile devices. Now those are commonly referred to as Apps. Even after almost a decade working in this field, having made many mistakes, paid for them monetarily and with time, still making mistakes (but keep learning from them); I still feel very much like running a startup. Or in matter of fact multiple startups.

Now running a startup isn’t an easy task. Sometimes, I meet people who want to start up themselves, but they don’t realize how hard it can be. Especially when you are coming from a “stable” job environment – I could actually elaborate for hours on the interpretation of ‘stable’ in this sense, but that’s for another day. So coming from a “stable” job and starting your own company is a tough step. First and foremost, your usual monthly salary just disappears over night. You will not have any (significant) income for months and or even years. You are not gonna have free weekends, regular 9-to-5 working hours or even proper vacations. While most people actually know that, expect it and prepare for the monetary part of it in terms of financial projections and goals; I believe a lot of people significantly underestimate the psychological changes that come with it. It’s surely not for everyone.

Working at a startup comes with many freedoms and sometimes even perks (like soft drinks, snacks and cool people that we are working with). Over these past years we have had 29 people who worked with us and left on their own will – at S4BB Limited alone. As a matter of fact, I was just thinking about that today and it motivated me to write this post. In the age group of 25 to 40 year olds the average time an employee is with the same company is probably something like 2 years in Hong Kong. I don’t have any hard number on this, but this is what we have seen here and it is also what I know from other local companies. Compared to Germany’s employees who stay 11 years on average with the same company, Hong Kong has quite some turnover. To be honest, the labor market is much more competitive here allowing folks to change jobs easily. Plus, head hunters (some of which are close friends) are very talented and motivated to place the right person at the right company. I also have to say that people don’t quit jobs, they usually fire quit their managers. I didn’t just think of that, it’s commonly known and I do agree to that.

Being one of those “managers” myself who has been fired by quite a number of employees, I have to admit that every single time I get fired, I feel devastated for at least a couple of days – depending on how close we were or at least I thought we were. I believe one of the main differences between working in a large corporation and in a startup is about processes and projects. In large corporations you usually have a bigger project that lasts for a couple of months or a duty that is rather constant or something similar. Startups are small, very small, that means they don’t have any manpower to waste at any given time during the day. This also results in many tasks and duties being shifted around on very short notice. To someone coming from a large corporation and now working at a startup that can be frustrating as one might not be able to see the bigger picture with constantly changing tasks or it can be motivating and even challenging. You just have to be the right person for it. Whether you run the show or you help run it.

The only big advantage startups have over established companies is speed:

  • Speed in developing and launching new products and services.
  • Speed in reacting to changes in the business environment.
  • Speed in decision processes that are necessary to be made quickly.
  • Speed in experimenting, failing and succeeding.

At my uni graduation ceremony the rector was holding a speech and I only remember one thing he said that: “In this new world not the big fish eat the small ones. The quick fish eat the slow ones.” You could also say it the Bruce Lee way: “Be water, my friend.

We never stop learning and should always be prepared for change – every day – and then embrace it.

Hong Kong’s Not So Rich People

Just some days ago during an event at Cyberport, Hong Kong’s Tech Hub, I took a look out the window and noticed that right downstairs a rich guy and a not so rich guy (should I call him poor?) are living.

I find it interesting that these kinds of things are defined somewhat differently in other countries like Germany for example 🙂

P.S.: This post is more in the non-sense category, but after all the ‘serious’ stuff recently, I wanted to share another view about live in Hong Kong. By the way, none of the cars are mine and I am also not living there unfortunately. The houses are small, but they are bloody expensive.

Storing Large Amounts of Video Data Efficiently, Redundantly and Quickly Accessible

I am writing this down because I read about 50 blog articles and how tos on the web regarding this topic. Yet, I couldn’t find any good advise that fit my video data storage issue.

Like I mentioned before, I am also involved in Slate Takes Limited – a boutique video production company based out of Hong Kong. Slate Takes is currently finishing up the last touches of a 42 webisode travel series for The Vagabond Project (more info and the official trailer at their Vagabond Blog), working with clients like Red Bull at events, assisting in shooting movies, and helping international brands to turn their vision onto moving pictures. As all of our work is done at least in Full HD, you can imagine that there is a lot of data that needs to be handled, stored and backed up.

In matter of cost efficiency, nothing beats magnetic drives — still. The difference is not as big as it was some years ago, but it is still there. However, such “tapes” don’t really perform that well. Working video projects is not just about saving the final video somewhere. It is all about storing the “raw data” that comes from the camera consistently and easily accessible. If you need one day to find a tape, copy data from that tape to your computer, and then find out that the tape was damaged towards the end of it, you are f***ed. Yes, tapes do definitely provide a longer durability than hard drives or CDs, but in order to achieve redundancy, you would need to have at least two copies of each tape. We don’t even want to think about the speed if you need to access data quickly and get things done.

I have always been a fan of hard drives and RAID systems in general (about 8 years ago I had a server with 1 TB storage at home — I thought I’d never get that filled up!). We have been using a 4-bay Synology NAS loaded with 4x 2TB hard drives for about a year now. Running in RAID 5 they provide some data protection in case one hard drive has an hardware defect. On top of that, the system comes with a 1 GB/s LAN connection which allows fairly quick access. In fact, it is possible to edit videos using Final Cut Pro 7 on Mac connected to the NAS via a GigaBit switch — of course not as fast as accessing the data on a local hard drive nor on a local SSD drive; but it works. If that’s not fast enough, the GB/s connection allows quick data transfer to your local machine.

Now, one year after its incarnation, we had to look into expanding our storage again. 8 TB were filled up (from which about 6 TB were usable due to RAID5) and data kept flying in. All options were on the table, but I decided to stick with Synology and only upgrade to a 12-bay NAS which can easily be extended by another 12 bays to store up to 72 TB. Synology’s “DiskStation” software is easy to use, yet quite feature rich and low in maintenance. Furthermore, the same partitions that had been created on the first 4 TB could be kept; making the transition easier. The new setup looks like this:

The great thing is: this unit even has 2x 1 GB/s LAN connections which allows more than one user to access files with maximum data transfer rates. Two additional 2 GB hard drives were installed to increase our storage by 50%. The transition to the new system was pretty straight forward if you follow Synology’s Migration How To. This setup allows us now to expand the NAS by up to 6 hard drives at any time without worrying about needing to expand it to fit more drives. Now the “expansion process” is running and from the progress it shows so far, it might run for a while.

In case you are looking for a similar setup, here a quick wrap-up of the costs:

We will see how long this setup will last. At least for now, it comforts me that there is gonna be at least a year or two until we have to think about an alternative storage system. For now, this should be sufficient and price-wise it was well, required.

Building a Company Targeting Consumers versus Targeting Businesses – From an Online / Social Media Perspective

A couple of years ago when we started building mobile apps for BlackBerry® smartphones the majority of BlackBerry® users were business people. They used their devices because they got them from their employer. The biggest BlackBerry® app buyer group back then were businesses who wanted to improve their business processes and bring more functionality to mobile. Obviously, we could have gone the way of building applications and services for those companies. As we weren’t the best B2B sales people, we were in not in the best physical location (Kunming, PRC) to meet up with business decision makers and we couldn’t be bothered to nurture business relations and attend ‘networking events’; we chose the path of targeting consumers directly. In a sense, it’s more fun anyway.

By targeting consumers we have to focus on moving at a faster pace. We need to bring updates of products very frequently, we need to publish new products often and we have to hit the sweet spot of what people really want, and so on. On top of that we have to establish a proper relationship with our users. This includes multiple communication channels like email, direct instant messaging, Twitter, Facebook, etc. Since the launch of S4BB’s Facebook page back in October 2011, we could grow our fan base to over 120,000 fans in less than 6 months. Most recently, we are pushing this a bit more and are seeing tremendous interest. To our marketing interns: Great job so far!

Building a huge fan base is usually one of the main goals of consumer companies when it comes to social media like Facebook for example. Every marketer you might want to hire, keeps telling you: you need a huge base (newsletter subscribers, Twitter followers, Facebook fans, feed subscribers, etc.) and later on you just need to “activate” them to sell products and services to them. That does sound good in theory, but let’s be honest here – what’s happening in the real world? In fact most companies spent a lot of money on building up huge online / social media properties with Facebook pages and communities way bigger than our S4BB Apps. Yet, most of those brands utilize their social media presence in the traditional “one-way direct marketing” kind of sense by kicking out a discount voucher once a week and publishing news about their products. Hoping these initiatives pay off. Yes, we do that too and yes, everything can be tracked to ensure proper ROI. But the truly big thing is: social media actually allows us to have a direct bi-directional communication with our brand’s audience. “Social Media Gurus” can talk you to sleep about that topic. Yet, nobody has an answer on how to manage bi-directional conversations with hundreds of thousands of fans and followers who have different views about different topics, products; speak different languages, live in different time zones and countries, and could all talk at the same time — to you.

At S4BB Ltd. and Skylab Mobilesystems Ltd. we are building products for consumers in a large scale. We target global audiences in any country in the world with any mobile platform they prefer. At TreeCrunch Ltd. we are actually building a platform for companies which have huge online properties with many customers, followers, fans; simply users, I like to call them. TreeCrunch’s direct customer-engagement platform allows us to overcome these challenges in communicating with our audiences at S4BB and Skylab, but it also allows any brand like yours to do the same. Is your company up for that?

More to come. Stay tuned and subscribe to my RSS feed.

Related posts:

Scaling a Web Server to Serve 165 Million Ad Requests per Day


I am usually not the kind of guy who likes to boast with numbers, but in this case I believe it helps to put things into perspective. For over seven years I am in the mobile app business targeting consumers directly. Over the years we have created thousands of products, most of them paid, some completely free and some with ads.

For years, I did not believe in monetization of apps through advertising. Even today, I am still very skeptical about that because you need to have A LOT of ad requests per day so that any significant income can be generated from that. At the moment, our few (around 10) mobile applications that sport ad banners, generate up to 165 million (165,000,000) ad requests per day. That makes up to 5.115 billion ad requests per month. If you compare that to a fairly large mobile advertising company like Adfonic with their “35 billion ad requests per month” (src: About Adfonic) we are doing quite alright for a small app company. We are using a mix of different ad networks depending on what performs best on which platform. Besides that we are breaking it down into country/region to use appropriate ad provider that is best for that region. Part of that also involves ad providers who do not support certain platforms natively. So built server-side components to handle such ad requests in a “proxy” kind of way that still allows us to get ads shown in apps on a certain platform that is not officially supported by the ad provider. Basically, we have a mini website that just shows an ad; and we can have hundreds of thousands of mobile devices accessing this website from all over the world at the same time.

Using cloud service providers like Amazon AWS, Rackspace, SoftLayer, Microsoft Azure, or others any can serve a virtually unlimited number of requests these day. It all depends on your credit card limit. There are obviously usage patterns of programs of applications and over the course of the day, we have ups and downs. For example, at midnight GMT most people all over the world like to use our applications and therefore they request more ads. Five hours later, we experience the lowest traffic. Depending on application store release schedules, promotions, featured listings, user notifications, external promotions like on blogs or elsewhere, unexpected sudden spikes in traffic can occur any time. “Automatic Scaling” in combination with “Load Balancing” seemed to be the magic solutions for this.

After months of running a bunch of server instances behind a load balancer we were quite happy with the performance. It was easy for us determine usage patterns and see how many servers we need to serve the average maximum number of requests without our service to fail. We didn’t think that much about cost-optimization because our cloud computing bills weren’t that high; so we didn’t really work on such auto scaling components. That had two main disadvantages: Firstly, we spent more that we needed to as we probably didn’t need half of the server that were running while we had low traffic. Secondly, we were not prepared for sudden massive spikes in traffic. With TreeCrunch on the other hand, we are building a scalable system from the ground up.

So early this week I took three hours and looked into how to implement such auto-scaling. Actually, it took me two hours to install the tools properly. However, as you can see in the above charts, the current systems works in a way that if we hit a maximum amount of average traffic per server, a new server is being created and added to our load balancer — it then immediately starts serving ads. If the average traffic falls below a certain threshold, a server is being terminated and therefore stops serving ads. To minimize server load, we run really tiny PHP scripts that are extremely optimized just for the purpose of requesting an ad the desired ad company and delivering such ad to the client (mobile app). As web server we are currently using lighttpd which is very lightweight indeed. Interestingly, we noticed that there is not really a problem with “normal” system resources to handle a lot of requests. Our CPU usage is fairly acceptable (constantly around 50%), we don’t need any hard drive space as we are just proxy’ing requests and we don’t even run out of memory. The first limitation one of our ad servers hits, is the maximum number of used sockets which is by default around 32,000 (or something like that) on Debian-based linux. That’s more or less an artificial limitation by the operating, but we didn’t play with adjusting that one yet.

Summary: In a fairly short period of time, we managed to set up a proper auto scaling policy that allows to scale up to virtually unlimited numbers of ad requests. With fairly low budgets companies nowadays can setup proper data centers, serve millions of users and maintain their infrastructure with a few very talented people and without purchasing any hardware that would be obsolete a year or two later.

I love the time I am living in. Every single day something new and exciting pops up.