I am writing this down because I read about 50 blog articles and how tos on the web regarding this topic. Yet, I couldn’t find any good advise that fit my video data storage issue.
Like I mentioned before, I am also involved in Slate Takes Limited – a boutique video production company based out of Hong Kong. Slate Takes is currently finishing up the last touches of a 42 webisode travel series for The Vagabond Project (more info and the official trailer at their Vagabond Blog), working with clients like Red Bull at events, assisting in shooting movies, and helping international brands to turn their vision onto moving pictures. As all of our work is done at least in Full HD, you can imagine that there is a lot of data that needs to be handled, stored and backed up.
In matter of cost efficiency, nothing beats magnetic drives — still. The difference is not as big as it was some years ago, but it is still there. However, such “tapes” don’t really perform that well. Working video projects is not just about saving the final video somewhere. It is all about storing the “raw data” that comes from the camera consistently and easily accessible. If you need one day to find a tape, copy data from that tape to your computer, and then find out that the tape was damaged towards the end of it, you are f***ed. Yes, tapes do definitely provide a longer durability than hard drives or CDs, but in order to achieve redundancy, you would need to have at least two copies of each tape. We don’t even want to think about the speed if you need to access data quickly and get things done.
I have always been a fan of hard drives and RAID systems in general (about 8 years ago I had a server with 1 TB storage at home — I thought I’d never get that filled up!). We have been using a 4-bay Synology NAS loaded with 4x 2TB hard drives for about a year now. Running in RAID 5 they provide some data protection in case one hard drive has an hardware defect. On top of that, the system comes with a 1 GB/s LAN connection which allows fairly quick access. In fact, it is possible to edit videos using Final Cut Pro 7 on Mac connected to the NAS via a GigaBit switch — of course not as fast as accessing the data on a local hard drive nor on a local SSD drive; but it works. If that’s not fast enough, the GB/s connection allows quick data transfer to your local machine.
Now, one year after its incarnation, we had to look into expanding our storage again. 8 TB were filled up (from which about 6 TB were usable due to RAID5) and data kept flying in. All options were on the table, but I decided to stick with Synology and only upgrade to a 12-bay NAS which can easily be extended by another 12 bays to store up to 72 TB. Synology’s “DiskStation” software is easy to use, yet quite feature rich and low in maintenance. Furthermore, the same partitions that had been created on the first 4 TB could be kept; making the transition easier. The new setup looks like this:
The great thing is: this unit even has 2x 1 GB/s LAN connections which allows more than one user to access files with maximum data transfer rates. Two additional 2 GB hard drives were installed to increase our storage by 50%. The transition to the new system was pretty straight forward if you follow Synology’s Migration How To. This setup allows us now to expand the NAS by up to 6 hard drives at any time without worrying about needing to expand it to fit more drives. Now the “expansion process” is running and from the progress it shows so far, it might run for a while.
In case you are looking for a similar setup, here a quick wrap-up of the costs:
We will see how long this setup will last. At least for now, it comforts me that there is gonna be at least a year or two until we have to think about an alternative storage system. For now, this should be sufficient and price-wise it was well, required.
Building a Company Targeting Consumers versus Targeting Businesses – From an Online / Social Media Perspective2012-03-28, on 13:56 | In work | No Comments | Dieser Artikel in Deutsch
A couple of years ago when we started building mobile apps for BlackBerry® smartphones the majority of BlackBerry® users were business people. They used their devices because they got them from their employer. The biggest BlackBerry® app buyer group back then were businesses who wanted to improve their business processes and bring more functionality to mobile. Obviously, we could have gone the way of building applications and services for those companies. As we weren’t the best B2B sales people, we were in not in the best physical location (Kunming, PRC) to meet up with business decision makers and we couldn’t be bothered to nurture business relations and attend ‘networking events’; we chose the path of targeting consumers directly. In a sense, it’s more fun anyway.
By targeting consumers we have to focus on moving at a faster pace. We need to bring updates of products very frequently, we need to publish new products often and we have to hit the sweet spot of what people really want, and so on. On top of that we have to establish a proper relationship with our users. This includes multiple communication channels like email, direct instant messaging, Twitter, Facebook, etc. Since the launch of S4BB’s Facebook page back in October 2011, we could grow our fan base to over 120,000 fans in less than 6 months. Most recently, we are pushing this a bit more and are seeing tremendous interest. To our marketing interns: Great job so far!
Building a huge fan base is usually one of the main goals of consumer companies when it comes to social media like Facebook for example. Every marketer you might want to hire, keeps telling you: you need a huge base (newsletter subscribers, Twitter followers, Facebook fans, feed subscribers, etc.) and later on you just need to “activate” them to sell products and services to them. That does sound good in theory, but let’s be honest here – what’s happening in the real world? In fact most companies spent a lot of money on building up huge online / social media properties with Facebook pages and communities way bigger than our S4BB Apps. Yet, most of those brands utilize their social media presence in the traditional “one-way direct marketing” kind of sense by kicking out a discount voucher once a week and publishing news about their products. Hoping these initiatives pay off. Yes, we do that too and yes, everything can be tracked to ensure proper ROI. But the truly big thing is: social media actually allows us to have a direct bi-directional communication with our brand’s audience. “Social Media Gurus” can talk you to sleep about that topic. Yet, nobody has an answer on how to manage bi-directional conversations with hundreds of thousands of fans and followers who have different views about different topics, products; speak different languages, live in different time zones and countries, and could all talk at the same time — to you.
At S4BB Ltd. and Skylab Mobilesystems Ltd. we are building products for consumers in a large scale. We target global audiences in any country in the world with any mobile platform they prefer. At TreeCrunch Ltd. we are actually building a platform for companies which have huge online properties with many customers, followers, fans; simply users, I like to call them. TreeCrunch’s direct customer-engagement platform allows us to overcome these challenges in communicating with our audiences at S4BB and Skylab, but it also allows any brand like yours to do the same. Is your company up for that?
More to come. Stay tuned and subscribe to my RSS feed.
I am usually not the kind of guy who likes to boast with numbers, but in this case I believe it helps to put things into perspective. For over seven years I am in the mobile app business targeting consumers directly. Over the years we have created thousands of products, most of them paid, some completely free and some with ads.
For years, I did not believe in monetization of apps through advertising. Even today, I am still very skeptical about that because you need to have A LOT of ad requests per day so that any significant income can be generated from that. At the moment, our few (around 10) mobile applications that sport ad banners, generate up to 165 million (165,000,000) ad requests per day. That makes up to 5.115 billion ad requests per month. If you compare that to a fairly large mobile advertising company like Adfonic with their “35 billion ad requests per month” (src: About Adfonic) we are doing quite alright for a small app company. We are using a mix of different ad networks depending on what performs best on which platform. Besides that we are breaking it down into country/region to use appropriate ad provider that is best for that region. Part of that also involves ad providers who do not support certain platforms natively. So built server-side components to handle such ad requests in a “proxy” kind of way that still allows us to get ads shown in apps on a certain platform that is not officially supported by the ad provider. Basically, we have a mini website that just shows an ad; and we can have hundreds of thousands of mobile devices accessing this website from all over the world at the same time.
Using cloud service providers like Amazon AWS, Rackspace, SoftLayer, Microsoft Azure, or others any can serve a virtually unlimited number of requests these day. It all depends on your credit card limit. There are obviously usage patterns of programs of applications and over the course of the day, we have ups and downs. For example, at midnight GMT most people all over the world like to use our applications and therefore they request more ads. Five hours later, we experience the lowest traffic. Depending on application store release schedules, promotions, featured listings, user notifications, external promotions like on blogs or elsewhere, unexpected sudden spikes in traffic can occur any time. “Automatic Scaling” in combination with “Load Balancing” seemed to be the magic solutions for this.
After months of running a bunch of server instances behind a load balancer we were quite happy with the performance. It was easy for us determine usage patterns and see how many servers we need to serve the average maximum number of requests without our service to fail. We didn’t think that much about cost-optimization because our cloud computing bills weren’t that high; so we didn’t really work on such auto scaling components. That had two main disadvantages: Firstly, we spent more that we needed to as we probably didn’t need half of the server that were running while we had low traffic. Secondly, we were not prepared for sudden massive spikes in traffic. With TreeCrunch on the other hand, we are building a scalable system from the ground up.
So early this week I took three hours and looked into how to implement such auto-scaling. Actually, it took me two hours to install the tools properly. However, as you can see in the above charts, the current systems works in a way that if we hit a maximum amount of average traffic per server, a new server is being created and added to our load balancer — it then immediately starts serving ads. If the average traffic falls below a certain threshold, a server is being terminated and therefore stops serving ads. To minimize server load, we run really tiny PHP scripts that are extremely optimized just for the purpose of requesting an ad the desired ad company and delivering such ad to the client (mobile app). As web server we are currently using lighttpd which is very lightweight indeed. Interestingly, we noticed that there is not really a problem with “normal” system resources to handle a lot of requests. Our CPU usage is fairly acceptable (constantly around 50%), we don’t need any hard drive space as we are just proxy’ing requests and we don’t even run out of memory. The first limitation one of our ad servers hits, is the maximum number of used sockets which is by default around 32,000 (or something like that) on Debian-based linux. That’s more or less an artificial limitation by the operating, but we didn’t play with adjusting that one yet.
Summary: In a fairly short period of time, we managed to set up a proper auto scaling policy that allows to scale up to virtually unlimited numbers of ad requests. With fairly low budgets companies nowadays can setup proper data centers, serve millions of users and maintain their infrastructure with a few very talented people and without purchasing any hardware that would be obsolete a year or two later.
I love the time I am living in. Every single day something new and exciting pops up.
Though, I don’t find much time to write articles these days, I believe I should write down my experiences as long as I still remember them properly. My fellow regular readers might have noticed a slight change in logos on the right sidebar which now sports the TreeCrunch logo; a company I co-founded a couple of months ago. In a nutshell, TreeCrunch is a social, viral, open-ended customer engagement platform with an incredible potential to influence companies and society.
From the beginning, we wanted to form a company that accomplishes four major goals:
- Build a successful tech company with global reach based out of Hong Kong.
- Help companies understand their social media audience and improve their business.
- Create a workplace where coders are kings and everyone loves to come to work.
- Expand as quickly as possible by raising funds internally and externally.
Building a tech company in Hong Kong is not easy. Yes, infrastructure is incredibly good — super fast non-firewalled Internet connection (that’s actually all a hacker needs), according to OECD most economic freedom, considerably low tax rates, access to all kinds of conveniences in life, a very good life style and any kind of beer your can imagine (important for us Germans) over one hundred “AAA” rated beaches. Yet, there are disadvantages and obstacles to overcome: recruiting of the best programmers available (remember, we are in competition with banks, hedge funds and property companies that are loaded with cash), high cost of lodging, cultural differences (to most uni grads is not as “cool” to work at a startup as it is silicon valley for example).
Helping companies understanding their social media audience is not the big problem. Our technology is superior, our prototypes are working, our first clients have committed to enjoy our services and work with us; growing with us.
Creating a great workplace for hackers is also the easier part for us. Over the past years we have been gathering a lot of information about how to make a programmer’s life fulfilling and fun. It can be quite boring when you are cramped in a 2 sqm area behind a tiny desk right next to the guy who showers once a week. Let’s start with small things like free soft drinks and snacks, free Friday lunches, air hockey tables, going over to purposely not enforcing too harsh deadlines (taking off pressure) and ending up in private rooms for each developer with two 23 inch (or bigger) monitors and the fastest desktop computers money buy. It is a very “not typical HK style” workplace and it is obviously not the right place for everyone. We believe the ones who like to work without too much supervision, but a lot of responsibility and creative ownership – those will change the world and they will change other people’s worlds.
Expanding as quickly as we can is not easy either. With S4BB, Skylab and Slate Takes (the other logos on the right) we have always pursued the way of self-sustaining growth without external funding. So for years now, our strategy has been exclusively: get profitable first; then spend money. That came with a lot advantages like not being in debt, having complete control over the direction the company and its products need to go. For example, a couple of years back we decided not to do any contractual mobile development work with S4BB and Skylab which obviously had a negative effect on cashflow in the short term. We decided to spend our time (==money) to build our own products that became assets of our business and have helped us a lot over the long term to create sustainable constant income to fund the creation of an awesome hacker workplace. On top of that, I don’t like to bring someone else’s dream to life.
With TreeCrunch on the other hand, we are going down the typical “Silicon-Valley-Style Tech Startup Road” with a slight twist. We are three co-founders that invested their own money, have a unique vision, developed superior technology, come with over 30 years of combined industry experience and (most importantly) we have run companies before; without going bankrupt. Even though we had enough ‘internal’ funding that we wouldn’t need to raise more, we are still doing it. Everyone knows this, but sometimes it has to be spoken out loud: “You got to raise money when you don’t need it.” That is one of the reasons why we are about to close our first pre-seed funding round raising our cash reserves by about 50%. Furthermore, TreeCrunch just got awarded the Cyberport Creative Micro Fund which comes with a grant of HK$100k.
With TreeCrunch we joined several programs so far, these are two:
Fund raising in Hong Kong is an interesting challenge. There are plenty of semi-governmental funds (like CCMF), fully covered government schemes as well as private companies (like Microsoft) with their programs. These are great opportunities which we will look and enter into when it makes sense. There are also a lot of wealthy people and families around who made a fortune with properties, stocks and other businesses. The usual way of investing their wealth is not related with startups – and the sub-group of tech-related startups is even further away from that.
For example, you won’t believe how incredibly hard it seems to be to set up a simple thing like a “shareholder’s agreement”. In the U.S. there are templates for this, you can go to almost any lawyer in the bay area and you will get a template for a couple of bucks or shares. For those who don’t know: Hong Kong’s economy is built on two major pillars: the property and the stock market. Other major factors of Hong Kong’s economy include the financial industry, import/export and tourism/retail. Then, for quite a while there is not much coming along the list of important industries and at the further end of it some creative industries like movie production, media or IT are popping up. Hong Kong is an incredible place for business and life – yet it surprises me again and again that there are not that many companies actually creating assets like intellectual property for example. We want to create a successful tech company with global reach and along the way help transforming Hong Kong into a more diversified economy and create long-term highly qualified jobs.
We know that our biggest assets are our co-workers and we hire only the best we can find. This is where we put our money and it will help us fulfill our dreams and help our customers to solve their problems.
More information about TreeCrunch Limited: www.treecrunch.com
I like those websites that state very prominently that their site is secure, yet it is not.
I know it says that I am browsing their secure “server” which might not necessarily mean that the connection to that server is secure. However, a standard HTTPS connection would have been enough for me. Lucky me, I didn’t need to pay for that T-Shirt so no credit card information was sent anywhere. Anyone who wants to get one too, can sign up at DevCon Asia.
The feature list of Mac OS X Lion sounds really good to me. When it was released some days ago, I got so excited that I wanted to get it right away. But I had to hold off until the weekend! As it is Friday evening right now, I got right on it, paid the lousy $30 and am downloading right now. This is gonna be an even more exciting weekend. Kozzi and his Lion are gonna have a dance! 😎
After having been invited by like 20 friends to Google+ and every time I tried, the Google+ website said: “New submissions are not accepted at the moment.” or “Submissions closed.” or something like that — I nearly gave up. Today, I gave it another shot, clicked the join link and actually got in there. It made me feel “special” somehow because I finally got in and now have a Google+ account. That was probably intended by Google anyway (a common method used by invitation only conferences, member clubs, conferences to raise the awareness of something usually ordinary). The disappointing at the sign up process though, was this one:
Especially the “You either share your photos or you stay the **** out of Google+” mentality is surprising. Well, I think I don’t have any photos in Picasa, so I didn’t have a problem with allowing that 🙂
You can find me here: Kozen on Google+
The first Amazon order arrived and now I can actually start reading programming books. No worries, I got some
interesting ones marketing books as well 🙂
P.S.: I hope that lifts us out of the “dummy” category.
I felt a bit like back in Kindergarten where I was encouraged to draw lots of funny pictures with pencils and crayons. (Yes, socialist Eastern Germany had colored pencils too!) Today I could really leverage from those years of experience and bring in all my creativity to colorize some squares on a sheet of paper. One of the most important questions these days had to be answered: What color pattern will the new office floor have? Nearly everyone participated and a couple of minutes later we had some funky sketches flying around:
Yes, at a workplace where nearly everything is done digitally there is still space for some ‘offline’ work.
We have some free snacks in our office and due to incredible demand – we are mostly stocked with instant noodles, chips, chocolate and nuts. For drinks we have coffee, tee (black, red, green, flower, power), Coke, Pepsi, milk, ice tea. Here a photo one part of our cupboard with some instant noodles. Got to keep the coders happy! (and everyone else too :))
What snacks do you have?