Saturday, October 13, 2012

myVidster and personal update

Greetings Vidsters!  This is an update on whats been going on with myVidster and myself.  Most of my work on myVidster has been on the backend.  I recently retired a server and replaced it with a larger one.  This was needed to handle the site's growth.  To give you an idea of myVidster's growth here are some stats.

Pageviews (Google Analytics)
August: 162 million
September: 194 million

I have also spent a lot of time on mobile app development and manage to get a 4.0 rating on Google Play!  I know that iPhone users want a native app as well, but I am still undecided on if its worth my time developing a native app and take the risk of Apple rejecting it.  For now use the web app, its has all the same functionality that the native would have with the exception that you cannot download videos.

For personal, I decided to leave the boring Midwest and moved myVidster and myself to Austin TX.  Perfect timing because ACL is this weekend!  Expect to find me at local tech and start-up happy hours and events.

What to expect in the future for myVidster?  Currently I do not know, now that I have a stable mobile app and I have scaled out the site for future growth I now have the time and capacity to brainstorm on new and exciting ideas for myVidster.  If you have any suggestions feel free to contact me or post a comment.

Take care and keep collecting!

Wednesday, September 5, 2012

myVidster mobile app support reloaded!

Greetings Vidsters, I have spent the last two months improving mobile support for both the Android and iPhone/iOS devices.  I will admit that the first version of the mobile app had its laundry list of issues and user experience was OK at best.  Keep in mind that it was my first attempt at developing a mobile app and my goal was to get a 'working' version out just to see if I could do it.  Now that I have gained some experience, I decided to resume development with the goal of creating an app that you will want to use everyday.  Here are the goals I set for myself.

Only display mobile compatible videos
One of the biggest challenges was filtering out the video bookmarks that would not play on your mobile device.  To tackle this issue I had to create a MySQL memory table that keeps tracks of the video bookmarks that are mobile friendly (encoded in h.264).  This table is used to display only the mobile friendly videos on the homepage and in your subscriptions.  Please note that this filtering does not take place when running a search or when you navigate through a users video collection.  To enable filtering go to your setting screen in the app.

Improve the user experience
If you have been a active user of the mobile app then you already know that the user experience (UX) needed an overhaul.  With the help of my friend Ryan, we went though the interface of the app to make navigation more intuitive by adding left/right swipe navigation and redesigned the video display screen.  Features like video history was added and I did some changes to the css to make the app more attractive.

Improve performance and stability
Load times for displaying a video bookmark was around 3-10 seconds which is unacceptable for any web or mobile app.  With the used of smart caching and code optimizations load times have been reduce to 2-3 seconds.  Can I get loads times to under a second?  As I gain more experience with mobile development I should be able to reduce the loads times in future updates.  As for stability that is a no brainier, if the app crashes for any reason, I would track down the issue and fix it!

Where can I get this wonderful app?
For Android you can install it from Google Play or you can download APK file from us.

For iPhone please use our web app.  When Apple decides to relax their submission policies then I will submit the app to their app store.

That is all for now, take care and keep collecting!

Monday, July 2, 2012

July 16th, myVidster's Day of Shameless Promotion!

Greeting Vidsters!  myVidster has come a long way from its humble beginnings as a simple video bookmarking service to one of the largest social media networks on the web today.  myVidster reaches over 3 million unique visitors a month generating over 100 million page views!  The growth of the myVidster community was not due to a post on a popular blog or by the mainstream media.  Its success was from YOU.  You told your friends about myVidster, shared your favorite video bookmarks on Facebook and Twitter, provided feedback and help financially by upgrading to our PRO account.

I know you have given a lot and I am thankful for this, but I would like to ask for one small favor.  On Monday, July 16th I will be launching promotional campaign to inform the rest of the world of our wonderful service.  My issue is that myVidster's entire staff consist of one person who is myself.  I am asking you, the myVidster community to set aside of few mintues of your busy day to help in the promotions.

If you want to help this is what you can do.
  1. Follow us on Twitter @myvidster.
  2. Install and rate the Android myVidster App on Google Play.
  3. Share myVidster on your favorite social networks (Facebook, Twitter, Google+).
  4. Tell the world why you love myVidster on your Tumblr/Blogger/Wordpress.
  5. Add our video sidebar widget on your Tumblr/Blogger/Wordpress.
  6. Make a 60 second video about myVidster.
  7. Tell your friends about myVidster over beers.
  8. Be creative and have fun with it!
Some would ask, so what do I get in return for being your marketing rep for the day?  For one, I will be eternally grateful and if we was to cross paths I would offer to buy you a beer :).  If you are promoting myVidster to a large audience and you can provide proof of your work please contact me and I will upgrade your account to our PRO service for free.

So mark your calendars for this day, I will be post announcements on a few days in advance as a friendly reminder.

Thank for all the love and support.  Keep collecting!

Wednesday, May 30, 2012

myVidster Embraces NoSQL

If you are a hardcore web techie you have seen the word NoSQL toss around.  In a nutshell its an alternative database management system.  Most websites (including myVidster) are build with mySQL, a traditional relational database management system (RDBMS).  For the most part mySQL is one of the best database solutions out there and will meet most (if not all) of your data needs.

As myVidster user base grows so does its need for more and more data.  When your data gets into millions of stored rows and you are trying to insert 3000+ rows of data every 2 minutes, mySQL faults begin to show.

In my experience mySQL does everything well expect for inserting large amounts of data in a given batch job and your queries can take a performance hit when you are have to sort large amounts of user data.  Most developers will agree with my issue with inserts, but some will argue that proper use of indexes will solve your performance issues when sorting.  I beg to differ.

Lets say you want to sort all user notification by date and return the last 5.  You will put an index on both the user id and the post date.  If the users has 5000 notifications, mySQL will run a filesort on the post date index (all 5000).  Disk I/O starts to become an issue when you are auto refreshing user's notification every minute.  Now if mySQL provided sorted indexes, then fetching the last 5 rows by date would not require a filesort, but sadly mySQL does not provide this feature.

I knew I have reach the limits of what mySQL can do so I decided to look into MongoDB. MongoDB is a NoSQL document oriented storage database.  During my testing MongoDB could insert documents (there like rows in a table) at a rate of 1000+/sec.  That solved my first problem, what about querying large data sets?  MongoDB provides sorted indexes and as long as you define the sort order of your indexes correctly (ascending or descending) MongoDB will not need to perform an nscanned (its like mySQL filesort) on the user's data.  I am also using MongoDB for backing up and achieving data as well.

Am I planning on replacing mySQL with MongoDB?  For now my answer is no, MongoDB does not support table linking and the code rewrite would be massive and the performance gain might not be worth it.  I do see myself using MongoDB for new features that I have in the pipeline.

Well, I think that is enough geek talk for now, take care and keep collecting!

Monday, April 30, 2012

Image Server Revisited

Since my last post I have made important changes to how images (mostly thumbnails) are serve on myVidster.  Stated in my last post I divided the image workload between the new image server and the master server.  I really did not like this setup because the master server should not be wasting disk I/O on image hosting.

My first attempt was to setup the image server to handle the entire load.  This solution would have work if I was running RAID on the server, but I was not.  It only took a few minutes before the server's hard disk was overwhelm.  When you run into the dreaded disk I/O bottleneck you have two options which are to upgrade your hardware (SSD or RAID) or memory caching.

If you have been keeping up with my blog, you will know that I try to use in memory caching whenever possible so I did some research and learn about using a reverse proxy for image caching.   For the reverse proxy I decided to give Varnish Cache a try, which was recommended by a developer/friend.

Setting up Varnish has straight forward the only catch was that I needed to disable cookies (remove req.http.Cookie) so that the images were always cached.  Technicality the image server is now a web application accelerator with the task of caching and serving images.

As for performance, currently I am getting a hit rate around 85% with 400-500 requests per second.  CPU load hovering around 10% which gives the server lots of room for handling more traffic when needed.  Overall I am very pleased with Varnish's caching performance and would recommend this solution to those who want to accelerate image serving and to reduce disk I/O.

For the curious here is a list of memory caching solutions that myVidster uses.
Alternative PHP Cache (APC) opcode caching for PHP
Memcached for user session caching

Take care everyone and keep collecting!

Tuesday, April 24, 2012

I'm Back, Actually I Never Left

Greetings Vidsters and guests! Here is a development update for the curious.

I recently added a new server to the family.  This one will act as a image server to offload some of the image hosting that the master server has been handling.  Also on my master server I had an under utilize RAID disk, so I move the most of the images (ie video thumbnails) to RAID.  These updates have reduce server load during peak hours by around 30%.

Also in the works is an automated system for removing video bookmarks that no longer play.

On a personal note, I was in San Fransisco for 2 weeks for business and I love it!  What I love the most was not the city, but the people.  They are diverse, creative and all around good people.  Living in the Midwest you get good people but can fall short on the diversity and creativity.  Not trying to bash my own home, oh wait I will ;).  Anyway there is a possibility that I might move out there, if I do watch out, because I am bringing a storm with me!

Take care and keep collecting

Monday, March 5, 2012

New Feature: myVidster Group Pages!

Greetings Vidsters!  It has been over a month since my last blog post and during that time I have been collecting your suggestions and feedback.  Most of your feedback revolve around the theme bring users together.  Users requested chat rooms, a discussion board and I have received emails of users seeking other users of similar interests.

While setting up chat rooms or a discussion board would be easy to do, I did not feel that it would integrate well with myVidster.  This lead me to a group page solution.  Users can create a group page on any subject they are interested in and use that page to attract and build a community.  Here is break down of the features.

Group Pages, one page to rule them all!

Wall (Group homepage)
Members can post to the wall and if they desire provide link(s) to myVidster video bookmarks to share to the group.  When you provide the link(s), the thumbnail, title and description from the video bookmark will be added to the wall post. Replying to a wall post is also supported.

Shared Videos
Members can share a video channel to the group.  This is a easy way for members to automatically share videos to other members.  It is also recommended that you share a channel that contains videos that are directly related to the group.

When adding a feature it needs to be integrated if you expect users to use it!  Notifications are sent out to the members when a new wall post is created.  You can auto post a video bookmark to the group's wall by using the video bookmark's share button.  myVidster's search engine  now supports group search.  Select groups in the search drop down to perform a group search.

Missing Features
Currently the only administration task the group owner can do is delete member's wall posts.  Future updates will include giving group owner the ability to manage members and add moderators.  Users will also be given the option to add the group's shared videos to their video subscriptions.

For the curious I have created a test group page, feel free to join it.  I do encourage active users to create their own group page.  Well that is it for now, as always keep providing the feedback.

Wednesday, January 25, 2012

Site Optimization and the Law of Diminishing Returns

If you have reading my blog, site optimization has always been apart of my development.  Techniques like rewriting slow sql queries and in memory caching will boost the performance of your site and reduce the need of purchasing more hardware. 

Currently myVidster's growth is at 16% (Source: Google Analytics).  While this is great, I have to activity monitor the site to ensure that performance stays optimal.  Most pages like the homepage, video bookmark and user profile will load within a second even during peak traffic.  Performance starts to take a hit when viewing a collection homepage and user subscription page when the video bookmark count are in the 1k+ range.  In this case performance is reduce to 2-4 seconds.

In response I have spent the last few weeks combing though my code and doing some house keeping cleaning up my code and updating sql queries.  In the end I was only able to increase performance by 500-1000ms.  On the bright side I did reduce the memory consumption of my PHP processes :).

Now we enter the Law of Diminishing Returns.  Do I spend more time trying to optimize my code or do I upgrade or add new hardware?  I know I can only do so much optimization before I get to the point that the benefits no longer justify the time I am spending (time is money).  You also have side affects where an update intended for optimization breaks or limits site's functionality.

With that said, I will most likely add a new server to handle myVidster's growth.  I being a geek always enjoy adding a new server to the family.  Well that is all for now, take care and keep collecting!

Saturday, January 21, 2012

New Feature: Subscription Viewer

Yo dawg, I herd you like viewing your video subscriptions, so I put an subscription viewer in your chat bar so you can view your subscriptions while you watch videos.
I hope I got that meme right :).  As stated in the meme (or lack of), you can now view your subscriptions while you navigate myVidster!  Before I started developing this feature I had set 3 goals.  The viewer needed to have a simple user interface, fast loading and non obtrusive. Here is a summary of how I accomplished these goals.

User Interface:
The viewer looks a lot like the existing subscription page, but with fewer features (you cannot search/display videos by user).  Users are given the option to naviagate though their subscriptions by clicking the prev and next buttons.  The state of the viewer (what page its on) is maintain while the user navigates the site.  To notify the user of new videos collected by those they follow, the existing video bookmarks in the viewer will fade out and the new video bookmarks will fade in.

Fast Loading:
Smart caching is used to speed up load times and when the data needs refreshing the data is pulled from mysql memory tables.  Other features like minify html/js/css are also used to minimize the 'payload' sent to the user's web browser.

Non Obtrusive:
By default the viewer will remain open and auto refresh is enabled.  While I feel these features are great I know some users would prefer not to use them.  I can see this being the case with users with low resolution monitors (especially older laptops) where the viewer is taking up too much space.  With that in mind, users can disable the keep open and auto refresh feature.

I hope you will enjoy this new feature as much as I enjoyed coding it :).  That is all for now and keep collecting!

Sunday, January 1, 2012

A geeky explanation on myVidster's growth

It's time to start the new year by geeking out!  December was a great month in terms of growth.  To give you an idea of myVidster's growth rate I ran a few queries for video bookmarking.

Video bookmarks for the last 3 months.
Oct: 300,460
Nov: 308,709
Dec: 357,149

Now lets take the daily average for December which gives us 11,520 video bookmarks a day!  We also have over 163,000 registered users.

To manage our growing community and to keep the site running smoothy I have added another server to the family :).  Currently myVidster operates on 4 servers, now lets break down what each server is doing.

Server1: Web Server
Server2: Database
Server3: Chat/Messaging
Server4: Search

The servers do other important tasks, but the ones listed above are their primary tasks.  I have also been hard at work with code optimization for the search, chat/messaging and database services.  I will like a make a side note that chat/messaging performance has been improve greatly, so please message away!

Well that is it for now, take care and keep collecting!