live from China::Hong Kong S.A.R. (中国::香港特別行政區)

Storing Large Amounts of Video Data Efficiently, Redundantly and Quickly Accessible

2012-05-08, on 23:08 | In work | 2 Comments | Dieser Artikel in Deutsch

I am writing this down because I read about 50 blog articles and how tos on the web regarding this topic. Yet, I couldn’t find any good advise that fit my video data storage issue.

Like I mentioned before, I am also involved in Slate Takes Limited – a boutique video production company based out of Hong Kong. Slate Takes is currently finishing up the last touches of a 42 webisode travel series for The Vagabond Project (more info and the official trailer at their Vagabond Blog), working with clients like Red Bull at events, assisting in shooting movies, and helping international brands to turn their vision onto moving pictures. As all of our work is done at least in Full HD, you can imagine that there is a lot of data that needs to be handled, stored and backed up.

In matter of cost efficiency, nothing beats magnetic drives — still. The difference is not as big as it was some years ago, but it is still there. However, such “tapes” don’t really perform that well. Working video projects is not just about saving the final video somewhere. It is all about storing the “raw data” that comes from the camera consistently and easily accessible. If you need one day to find a tape, copy data from that tape to your computer, and then find out that the tape was damaged towards the end of it, you are f***ed. Yes, tapes do definitely provide a longer durability than hard drives or CDs, but in order to achieve redundancy, you would need to have at least two copies of each tape. We don’t even want to think about the speed if you need to access data quickly and get things done.

I have always been a fan of hard drives and RAID systems in general (about 8 years ago I had a server with 1 TB storage at home — I thought I’d never get that filled up!). We have been using a 4-bay Synology NAS loaded with 4x 2TB hard drives for about a year now. Running in RAID 5 they provide some data protection in case one hard drive has an hardware defect. On top of that, the system comes with a 1 GB/s LAN connection which allows fairly quick access. In fact, it is possible to edit videos using Final Cut Pro 7 on Mac connected to the NAS via a GigaBit switch — of course not as fast as accessing the data on a local hard drive nor on a local SSD drive; but it works. If that’s not fast enough, the GB/s connection allows quick data transfer to your local machine.

Now, one year after its incarnation, we had to look into expanding our storage again. 8 TB were filled up (from which about 6 TB were usable due to RAID5) and data kept flying in. All options were on the table, but I decided to stick with Synology and only upgrade to a 12-bay NAS which can easily be extended by another 12 bays to store up to 72 TB. Synology’s “DiskStation” software is easy to use, yet quite feature rich and low in maintenance. Furthermore, the same partitions that had been created on the first 4 TB could be kept; making the transition easier. The new setup looks like this:

The great thing is: this unit even has 2x 1 GB/s LAN connections which allows more than one user to access files with maximum data transfer rates. Two additional 2 GB hard drives were installed to increase our storage by 50%. The transition to the new system was pretty straight forward if you follow Synology’s Migration How To. This setup allows us now to expand the NAS by up to 6 hard drives at any time without worrying about needing to expand it to fit more drives. Now the “expansion process” is running and from the progress it shows so far, it might run for a while.

In case you are looking for a similar setup, here a quick wrap-up of the costs:

We will see how long this setup will last. At least for now, it comforts me that there is gonna be at least a year or two until we have to think about an alternative storage system. For now, this should be sufficient and price-wise it was well, required.

Building a Company Targeting Consumers versus Targeting Businesses – From an Online / Social Media Perspective

2012-03-28, on 13:56 | In work | No Comments | Dieser Artikel in Deutsch

A couple of years ago when we started building mobile apps for BlackBerry® smartphones the majority of BlackBerry® users were business people. They used their devices because they got them from their employer. The biggest BlackBerry® app buyer group back then were businesses who wanted to improve their business processes and bring more functionality to mobile. Obviously, we could have gone the way of building applications and services for those companies. As we weren’t the best B2B sales people, we were in not in the best physical location (Kunming, PRC) to meet up with business decision makers and we couldn’t be bothered to nurture business relations and attend ‘networking events’; we chose the path of targeting consumers directly. In a sense, it’s more fun anyway.

By targeting consumers we have to focus on moving at a faster pace. We need to bring updates of products very frequently, we need to publish new products often and we have to hit the sweet spot of what people really want, and so on. On top of that we have to establish a proper relationship with our users. This includes multiple communication channels like email, direct instant messaging, Twitter, Facebook, etc. Since the launch of S4BB’s Facebook page back in October 2011, we could grow our fan base to over 120,000 fans in less than 6 months. Most recently, we are pushing this a bit more and are seeing tremendous interest. To our marketing interns: Great job so far!

Building a huge fan base is usually one of the main goals of consumer companies when it comes to social media like Facebook for example. Every marketer you might want to hire, keeps telling you: you need a huge base (newsletter subscribers, Twitter followers, Facebook fans, feed subscribers, etc.) and later on you just need to “activate” them to sell products and services to them. That does sound good in theory, but let’s be honest here – what’s happening in the real world? In fact most companies spent a lot of money on building up huge online / social media properties with Facebook pages and communities way bigger than our S4BB Apps. Yet, most of those brands utilize their social media presence in the traditional “one-way direct marketing” kind of sense by kicking out a discount voucher once a week and publishing news about their products. Hoping these initiatives pay off. Yes, we do that too and yes, everything can be tracked to ensure proper ROI. But the truly big thing is: social media actually allows us to have a direct bi-directional communication with our brand’s audience. “Social Media Gurus” can talk you to sleep about that topic. Yet, nobody has an answer on how to manage bi-directional conversations with hundreds of thousands of fans and followers who have different views about different topics, products; speak different languages, live in different time zones and countries, and could all talk at the same time — to you.

At S4BB Ltd. and Skylab Mobilesystems Ltd. we are building products for consumers in a large scale. We target global audiences in any country in the world with any mobile platform they prefer. At TreeCrunch Ltd. we are actually building a platform for companies which have huge online properties with many customers, followers, fans; simply users, I like to call them. TreeCrunch’s direct customer-engagement platform allows us to overcome these challenges in communicating with our audiences at S4BB and Skylab, but it also allows any brand like yours to do the same. Is your company up for that?

More to come. Stay tuned and subscribe to my RSS feed.

Related posts:

Scaling a Web Server to Serve 165 Million Ad Requests per Day

2012-03-14, on 15:17 | In work | 3 Comments | Dieser Artikel in Deutsch


I am usually not the kind of guy who likes to boast with numbers, but in this case I believe it helps to put things into perspective. For over seven years I am in the mobile app business targeting consumers directly. Over the years we have created thousands of products, most of them paid, some completely free and some with ads.

For years, I did not believe in monetization of apps through advertising. Even today, I am still very skeptical about that because you need to have A LOT of ad requests per day so that any significant income can be generated from that. At the moment, our few (around 10) mobile applications that sport ad banners, generate up to 165 million (165,000,000) ad requests per day. That makes up to 5.115 billion ad requests per month. If you compare that to a fairly large mobile advertising company like Adfonic with their “35 billion ad requests per month” (src: About Adfonic) we are doing quite alright for a small app company. We are using a mix of different ad networks depending on what performs best on which platform. Besides that we are breaking it down into country/region to use appropriate ad provider that is best for that region. Part of that also involves ad providers who do not support certain platforms natively. So built server-side components to handle such ad requests in a “proxy” kind of way that still allows us to get ads shown in apps on a certain platform that is not officially supported by the ad provider. Basically, we have a mini website that just shows an ad; and we can have hundreds of thousands of mobile devices accessing this website from all over the world at the same time.

Using cloud service providers like Amazon AWS, Rackspace, SoftLayer, Microsoft Azure, or others any can serve a virtually unlimited number of requests these day. It all depends on your credit card limit. There are obviously usage patterns of programs of applications and over the course of the day, we have ups and downs. For example, at midnight GMT most people all over the world like to use our applications and therefore they request more ads. Five hours later, we experience the lowest traffic. Depending on application store release schedules, promotions, featured listings, user notifications, external promotions like on blogs or elsewhere, unexpected sudden spikes in traffic can occur any time. “Automatic Scaling” in combination with “Load Balancing” seemed to be the magic solutions for this.

After months of running a bunch of server instances behind a load balancer we were quite happy with the performance. It was easy for us determine usage patterns and see how many servers we need to serve the average maximum number of requests without our service to fail. We didn’t think that much about cost-optimization because our cloud computing bills weren’t that high; so we didn’t really work on such auto scaling components. That had two main disadvantages: Firstly, we spent more that we needed to as we probably didn’t need half of the server that were running while we had low traffic. Secondly, we were not prepared for sudden massive spikes in traffic. With TreeCrunch on the other hand, we are building a scalable system from the ground up.

So early this week I took three hours and looked into how to implement such auto-scaling. Actually, it took me two hours to install the tools properly. However, as you can see in the above charts, the current systems works in a way that if we hit a maximum amount of average traffic per server, a new server is being created and added to our load balancer — it then immediately starts serving ads. If the average traffic falls below a certain threshold, a server is being terminated and therefore stops serving ads. To minimize server load, we run really tiny PHP scripts that are extremely optimized just for the purpose of requesting an ad the desired ad company and delivering such ad to the client (mobile app). As web server we are currently using lighttpd which is very lightweight indeed. Interestingly, we noticed that there is not really a problem with “normal” system resources to handle a lot of requests. Our CPU usage is fairly acceptable (constantly around 50%), we don’t need any hard drive space as we are just proxy’ing requests and we don’t even run out of memory. The first limitation one of our ad servers hits, is the maximum number of used sockets which is by default around 32,000 (or something like that) on Debian-based linux. That’s more or less an artificial limitation by the operating, but we didn’t play with adjusting that one yet.

Summary: In a fairly short period of time, we managed to set up a proper auto scaling policy that allows to scale up to virtually unlimited numbers of ad requests. With fairly low budgets companies nowadays can setup proper data centers, serve millions of users and maintain their infrastructure with a few very talented people and without purchasing any hardware that would be obsolete a year or two later.

I love the time I am living in. Every single day something new and exciting pops up.

Traveling the Silicon-Valley-Style Tech Startup Road

2012-02-12, on 13:51 | In work | No Comments | Dieser Artikel in Deutsch

Though, I don’t find much time to write articles these days, I believe I should write down my experiences as long as I still remember them properly. My fellow regular readers might have noticed a slight change in logos on the right sidebar which now sports the TreeCrunch logo; a company I co-founded a couple of months ago. In a nutshell, TreeCrunch is a social, viral, open-ended customer engagement platform with an incredible potential to influence companies and society.

From the beginning, we wanted to form a company that accomplishes four major goals:

  1. Build a successful tech company with global reach based out of Hong Kong.
  2. Help companies understand their social media audience and improve their business.
  3. Create a workplace where coders are kings and everyone loves to come to work.
  4. Expand as quickly as possible by raising funds internally and externally.

Building a tech company in Hong Kong is not easy. Yes, infrastructure is incredibly good — super fast non-firewalled Internet connection (that’s actually all a hacker needs), according to OECD most economic freedom, considerably low tax rates, access to all kinds of conveniences in life, a very good life style and any kind of beer your can imagine (important for us Germans) over one hundred “AAA” rated beaches. Yet, there are disadvantages and obstacles to overcome: recruiting of the best programmers available (remember, we are in competition with banks, hedge funds and property companies that are loaded with cash), high cost of lodging, cultural differences (to most uni grads is not as “cool” to work at a startup as it is silicon valley for example).

Helping companies understanding their social media audience is not the big problem. Our technology is superior, our prototypes are working, our first clients have committed to enjoy our services and work with us; growing with us.

Creating a great workplace for hackers is also the easier part for us. Over the past years we have been gathering a lot of information about how to make a programmer’s life fulfilling and fun. It can be quite boring when you are cramped in a 2 sqm area behind a tiny desk right next to the guy who showers once a week. Let’s start with small things like free soft drinks and snacks, free Friday lunches, air hockey tables, going over to purposely not enforcing too harsh deadlines (taking off pressure) and ending up in private rooms for each developer with two 23 inch (or bigger) monitors and the fastest desktop computers money buy. It is a very “not typical HK style” workplace and it is obviously not the right place for everyone. We believe the ones who like to work without too much supervision, but a lot of responsibility and creative ownership – those will change the world and they will change other people’s worlds.

Expanding as quickly as we can is not easy either. With S4BB, Skylab and Slate Takes (the other logos on the right) we have always pursued the way of self-sustaining growth without external funding. So for years now, our strategy has been exclusively: get profitable first; then spend money. That came with a lot advantages like not being in debt, having complete control over the direction the company and its products need to go. For example, a couple of years back we decided not to do any contractual mobile development work with S4BB and Skylab which obviously had a negative effect on cashflow in the short term. We decided to spend our time (==money) to build our own products that became assets of our business and have helped us a lot over the long term to create sustainable constant income to fund the creation of an awesome hacker workplace. On top of that, I don’t like to bring someone else’s dream to life.

With TreeCrunch on the other hand, we are going down the typical “Silicon-Valley-Style Tech Startup Road” with a slight twist. We are three co-founders that invested their own money, have a unique vision, developed superior technology, come with over 30 years of combined industry experience and (most importantly) we have run companies before; without going bankrupt. Even though we had enough ‘internal’ funding that we wouldn’t need to raise more, we are still doing it. Everyone knows this, but sometimes it has to be spoken out loud: “You got to raise money when you don’t need it.” That is one of the reasons why we are about to close our first pre-seed funding round raising our cash reserves by about 50%. Furthermore, TreeCrunch just got awarded the Cyberport Creative Micro Fund which comes with a grant of HK$100k.

With TreeCrunch we joined several programs so far, these are two:

Fund raising in Hong Kong is an interesting challenge. There are plenty of semi-governmental funds (like CCMF), fully covered government schemes as well as private companies (like Microsoft) with their programs. These are great opportunities which we will look and enter into when it makes sense. There are also a lot of wealthy people and families around who made a fortune with properties, stocks and other businesses. The usual way of investing their wealth is not related with startups – and the sub-group of tech-related startups is even further away from that.

For example, you won’t believe how incredibly hard it seems to be to set up a simple thing like a “shareholder’s agreement”. In the U.S. there are templates for this, you can go to almost any lawyer in the bay area and you will get a template for a couple of bucks or shares. For those who don’t know: Hong Kong’s economy is built on two major pillars: the property and the stock market. Other major factors of Hong Kong’s economy include the financial industry, import/export and tourism/retail. Then, for quite a while there is not much coming along the list of important industries and at the further end of it some creative industries like movie production, media or IT are popping up. Hong Kong is an incredible place for business and life – yet it surprises me again and again that there are not that many companies actually creating assets like intellectual property for example. We want to create a successful tech company with global reach and along the way help transforming Hong Kong into a more diversified economy and create long-term highly qualified jobs.

We know that our biggest assets are our co-workers and we hire only the best we can find. This is where we put our money and it will help us fulfill our dreams and help our customers to solve their problems.

More information about TreeCrunch Limited: www.treecrunch.com

Stocked up the library

2011-06-17, on 20:21 | In work | No Comments | Dieser Artikel in Deutsch

The first Amazon order arrived and now I can actually start reading programming books. No worries, I got some interesting ones marketing books as well 🙂

P.S.: I hope that lifts us out of the “dummy” category.

Creativity with Colored Pencils at Work

2011-05-06, on 17:59 | In work | 1 Comment | Dieser Artikel in Deutsch

I felt a bit like back in Kindergarten where I was encouraged to draw lots of funny pictures with pencils and crayons. (Yes, socialist Eastern Germany had colored pencils too!) Today I could really leverage from those years of experience and bring in all my creativity to colorize some squares on a sheet of paper. One of the most important questions these days had to be answered: What color pattern will the new office floor have? Nearly everyone participated and a couple of minutes later we had some funky sketches flying around:

Yes, at a workplace where nearly everything is done digitally there is still space for some ‘offline’ work.

Office Stock – What kind of snacks do you have in your office?

2011-03-17, on 23:23 | In work | No Comments | Dieser Artikel in Deutsch

We have some free snacks in our office and due to incredible demand – we are mostly stocked with instant noodles, chips, chocolate and nuts. For drinks we have coffee, tee (black, red, green, flower, power), Coke, Pepsi, milk, ice tea. Here a photo one part of our cupboard with some instant noodles. Got to keep the coders happy! (and everyone else too :))


(photo was taken behind a glass wall, with very low light on a BlackBerry Torch – sorry for the bad quality)

What snacks do you have?

China Copy – They can’t even take their own photos

2011-03-17, on 11:23 | In work | No Comments | Dieser Artikel in Deutsch

I just received a spam email to a non-public business email address that is forwarded to multiple people. The title of that email is: 專業..CCTV..DVR.. (Tel: 6063 7382) So obviously this was a spam email and everyone of us is getting those every single day. We can’t do much about it and still, sometimes we even open those spam emails and take a look into them. This one was quite surprising.

This is what shows in that spam email:

[…]
IT Solutions


[…]

There were also other images with CCTV equipment and stuff.

Now everyone who reads Joel On Software might have already noticed that the photo above somehow appeared somewhere before. The actual article from Joel back in December 2008 “The new Fog Creek office” showed that very photo – here is the ‘original’ one:


(Courtesy Joel Spolsky at FogCreek)

I doubt that Chinese factory actually obtained a license to advertise their products with that image. Darn, the photo is not even one that has been taken with one of their CCTV cameras 🙂

Freelancer.com – I tried to deal with their support

2011-02-09, on 10:30 | In work | 7 Comments | Dieser Artikel in Deutsch

Freelancer.com has the worst customer support ever
Yes, we use Freelancer.com for outsourcing projects. I purposely don’t publish the link here. For over six years, we have had a good relation ship with them; though we barely interacted with the people behind Freelancer.com. For some reason, one payment to Freelancer.com did not work because PayPal messed something up. This resulted in our PayPal account to be locked down (which we easily re-activated) and our Freelancer.com account to be suspended as well. Freelancer.com still gets US$510 from us, but we can not pay them as our account is still suspended. Therefore, I took the chance to consult their online support – and I am not the first person who did that. This is what came out:

Myla: Hi there, how may I help you?

HR Department: Hello Myla
HR Department: Please review ticket YHV-383840
HR Department: We are still waiting on our account to be re-activated.
HR Department: Our PayPal account is back working and we would like to settle our debt with Freelancer.com now.

Myla: I will be more than happy to assist you.
Myla: Please give me a minute while I check this for you.

HR Department: I will be leaving for a congress soon and therefore, I will not have much time to take care of this the rest of this week. So it would really be great if we can get this resolved now.

Myla: Thank you for your patience. The ticket that you submitted have been forwarded to our Tier2 department. Unfortunately we will not be able to view the updates or status of this ticket from our end. Kindly wait for the reply from that department for the resolution of this issue. We are hoping for your understanding.

HR Department: Myla, we just want to pay you.
HR Department: Please connect me with your superior. If I can’t get this resolved within the next 30 minutes I will have to close our company account. Which means, your company will loose $510.
HR Department: and future projects we were going to post. As you can see from our track record, we are a respectable service buyer and actually bring business to your company.

Myla: We undestand how the situation can be frustrating for you. That is exactly the reason why the ticket was forwarded to our Tier2 department for accurate resolution. You need to wait for there reply regarding this concern. Rest assure that we will follow on this.

HR Department: Yes, frustrating and extremely disappointing.
HR Department: If you can not help me. Please put me in touch with someone who can. A phone number would be great for example.

Myla: We do not have a voice support in our site. We need you to wait for the reply on the ticket that was already forward to our Tier2 department.

HR Department: Please provide me a form to close our account.

Myla: We understand that you want to close your account . However, you have to reinstate your account first before we can process your request.

HR Department: You are kidding.

Myla: I am afraid not. Do you have any other inquiry?

HR Department: Yes.
HR Department: 1) I want our company account re-activated now!
HR Department: 2) If this is not possible, I want to close it now!
HR Department: These are my inquiries. Please help me with those.

Myla: Please understand that we need you to wait for the reply from our Tier2 department to resolve this issue. We cannot just reactivate or close your account without following the protocol. We will appreciate if you try to understand this points.

HR Department: When will your Tier2 department be working again? They did not do anything yesterday …

Myla: We will follow up on you ticket for the resolution. Rest assure we will reply to your ticket as soon as possible.
Myla: Thank you for contacting Freelancer.com, please don’t hesitate to contact us again if you have any other questions or concerns.

Myla has left the conversation
Click here to leave a message

I understand, that it might be out of Myla’s control, but then it is a massive design failure in their support system. However, Myla should never ever just hang up on a customer without any confirmation that he is not satisfied with the result of the support. May be that’s just my understanding of customer support and may be that’s the way how customer support works in Australia or Sweden?

Did you have a bad experience with them too? Feel free to leave a comment below or tweet your experience to them @freelancer @geta@freelancer or about them #freelancer #getafreelancer. I did: @kozen23

299 Top Executives, 1 Chief Executive und 1 Koz

2010-07-05, on 17:52 | In in hong kong, work | 5 Comments | Dieser Artikel in Deutsch

So liest sich das offizielle Intro:

The Chief Executive, Mr Donald Tsang, today (June 22) hosted a cocktail reception to thank the foreign investors whose presence in Hong Kong is a testimony to its position as Asia’s international business and finance centre. The cocktail reception, attended by about 300 top executives, is an annual event to warmly welcome overseas, Mainland and Taiwanese companies that have set up or expanded in Hong Kong […]

So lief Kozzi’s Version ab:

Vor ziemlich genau 2 Wochen hatte der Kozzi mal wieder einen öffentlichen Auftritt. Da dieses mal der Chief Executive von Hong Kong Donald Tsang zu Speis und Trank einlud, hatte der Koz auch eine Ausrede, um sich mal nen ordentlichen Zwirn maßschneidern (oder mit 3 ‘s’?) zu lassen. Eigentlich bin ich ja überhaupt kein Freund von solchen Veranstaltungen. Ein Informatiker ist für sowas einfach nicht gebaut worden. Obwohl ich gehört habe, daß Special Ops Field Engineers auf alles vorbereitet sind.

Donald hat am Anfang eine kurze über weite Strecken langweilige Rede gehalten worin er z.B. meinte, dass die HK Regierung ca. 10 mal pro Woche verklagt wird und er das gut findet. Offener Markt, Rechtsicherheit usw. wollte er wohl damit hervor heben. Wie auch immer, ich habe mich an den Rotwein, Lachs und Kaviar gehalten. Achja, und die Peking Ente bitte nicht vergessen, da war nämlich ausschließlich die gute Haut am Start und es wurde einem von nem Koch direkt mundgerecht zusammengerollt.

Irgendwann zur Mitte der Veranstaltung war dann Photo-Runde. Ein bißchen angeschwipst habe ich es auch aufs Treppchen geschafft. Ich glaube aber, das leicht ist keinem aufgefallen… Wer findet den Kozzi mit den roten Wangen auf dem Photo? 🙂

ce-reception-2010-s4bb

Mehr zu dem Thema:

Next Page »

about | kozens blog