Since my last post I have made important changes to how images (mostly thumbnails) are serve on myVidster. Stated in my last post I divided the image workload between the new image server and the master server. I really did not like this setup because the master server should not be wasting disk I/O on image hosting.
My first attempt was to setup the image server to handle the entire load. This solution would have work if I was running RAID on the server, but I was not. It only took a few minutes before the server's hard disk was overwhelm. When you run into the dreaded disk I/O bottleneck you have two options which are to upgrade your hardware (SSD or RAID) or memory caching.
If you have been keeping up with my blog, you will know that I try to use in memory caching whenever possible so I did some research and learn about using a reverse proxy for image caching. For the reverse proxy I decided to give Varnish Cache a try, which was recommended by a developer/friend.
Setting up Varnish has straight forward the only catch was that I needed to disable cookies (remove req.http.Cookie) so that the images were always cached. Technicality the image server is now a web application accelerator with the task of caching and serving images.
As for performance, currently I am getting a hit rate around 85% with 400-500 requests per second. CPU load hovering around 10% which gives the server lots of room for handling more traffic when needed. Overall I am very pleased with Varnish's caching performance and would recommend this solution to those who want to accelerate image serving and to reduce disk I/O.
For the curious here is a list of memory caching solutions that myVidster uses.
Alternative PHP Cache (APC) opcode caching for PHP
Memcached for user session caching
Take care everyone and keep collecting!
Monday, April 30, 2012
Tuesday, April 24, 2012
I'm Back, Actually I Never Left
Greetings Vidsters and guests! Here is a development update for the curious.
I recently added a new server to the family. This one will act as a image server to offload some of the image hosting that the master server has been handling. Also on my master server I had an under utilize RAID disk, so I move the most of the images (ie video thumbnails) to RAID. These updates have reduce server load during peak hours by around 30%.
Also in the works is an automated system for removing video bookmarks that no longer play.
On a personal note, I was in San Fransisco for 2 weeks for business and I love it! What I love the most was not the city, but the people. They are diverse, creative and all around good people. Living in the Midwest you get good people but can fall short on the diversity and creativity. Not trying to bash my own home, oh wait I will ;). Anyway there is a possibility that I might move out there, if I do watch out, because I am bringing a storm with me!
Take care and keep collecting
I recently added a new server to the family. This one will act as a image server to offload some of the image hosting that the master server has been handling. Also on my master server I had an under utilize RAID disk, so I move the most of the images (ie video thumbnails) to RAID. These updates have reduce server load during peak hours by around 30%.
Also in the works is an automated system for removing video bookmarks that no longer play.
On a personal note, I was in San Fransisco for 2 weeks for business and I love it! What I love the most was not the city, but the people. They are diverse, creative and all around good people. Living in the Midwest you get good people but can fall short on the diversity and creativity. Not trying to bash my own home, oh wait I will ;). Anyway there is a possibility that I might move out there, if I do watch out, because I am bringing a storm with me!
Take care and keep collecting
Monday, March 5, 2012
New Feature: myVidster Group Pages!
Greetings Vidsters! It has been over a month since my last blog post and during that time I have been collecting your suggestions and feedback. Most of your feedback revolve around the theme bring users together. Users requested chat rooms, a discussion board and I have received emails of users seeking other users of similar interests.
While setting up chat rooms or a discussion board would be easy to do, I did not feel that it would integrate well with myVidster. This lead me to a group page solution. Users can create a group page on any subject they are interested in and use that page to attract and build a community. Here is break down of the features.
Wall (Group homepage)
Members can post to the wall and if they desire provide link(s) to myVidster video bookmarks to share to the group. When you provide the link(s), the thumbnail, title and description from the video bookmark will be added to the wall post. Replying to a wall post is also supported.
Shared Videos
Members can share a video channel to the group. This is a easy way for members to automatically share videos to other members. It is also recommended that you share a channel that contains videos that are directly related to the group.
Integration
When adding a feature it needs to be integrated if you expect users to use it! Notifications are sent out to the members when a new wall post is created. You can auto post a video bookmark to the group's wall by using the video bookmark's share button. myVidster's search engine now supports group search. Select groups in the search drop down to perform a group search.
Missing Features
Currently the only administration task the group owner can do is delete member's wall posts. Future updates will include giving group owner the ability to manage members and add moderators. Users will also be given the option to add the group's shared videos to their video subscriptions.
For the curious I have created a test group page, feel free to join it. I do encourage active users to create their own group page. Well that is it for now, as always keep providing the feedback.
While setting up chat rooms or a discussion board would be easy to do, I did not feel that it would integrate well with myVidster. This lead me to a group page solution. Users can create a group page on any subject they are interested in and use that page to attract and build a community. Here is break down of the features.
 |
| Group Pages, one page to rule them all! |
Wall (Group homepage)
Members can post to the wall and if they desire provide link(s) to myVidster video bookmarks to share to the group. When you provide the link(s), the thumbnail, title and description from the video bookmark will be added to the wall post. Replying to a wall post is also supported.
Shared Videos
Members can share a video channel to the group. This is a easy way for members to automatically share videos to other members. It is also recommended that you share a channel that contains videos that are directly related to the group.
Integration
When adding a feature it needs to be integrated if you expect users to use it! Notifications are sent out to the members when a new wall post is created. You can auto post a video bookmark to the group's wall by using the video bookmark's share button. myVidster's search engine now supports group search. Select groups in the search drop down to perform a group search.
Missing Features
Currently the only administration task the group owner can do is delete member's wall posts. Future updates will include giving group owner the ability to manage members and add moderators. Users will also be given the option to add the group's shared videos to their video subscriptions.
For the curious I have created a test group page, feel free to join it. I do encourage active users to create their own group page. Well that is it for now, as always keep providing the feedback.
Wednesday, January 25, 2012
Site Optimization and the Law of Diminishing Returns
If you have reading my blog, site optimization has always been apart of my development. Techniques like rewriting slow sql queries and in memory caching will boost the performance of your site and reduce the need of purchasing more hardware.
Currently myVidster's growth is at 16% (Source: Google Analytics). While this is great, I have to activity monitor the site to ensure that performance stays optimal. Most pages like the homepage, video bookmark and user profile will load within a second even during peak traffic. Performance starts to take a hit when viewing a collection homepage and user subscription page when the video bookmark count are in the 1k+ range. In this case performance is reduce to 2-4 seconds.
In response I have spent the last few weeks combing though my code and doing some house keeping cleaning up my code and updating sql queries. In the end I was only able to increase performance by 500-1000ms. On the bright side I did reduce the memory consumption of my PHP processes :).
Now we enter the Law of Diminishing Returns. Do I spend more time trying to optimize my code or do I upgrade or add new hardware? I know I can only do so much optimization before I get to the point that the benefits no longer justify the time I am spending (time is money). You also have side affects where an update intended for optimization breaks or limits site's functionality.
With that said, I will most likely add a new server to handle myVidster's growth. I being a geek always enjoy adding a new server to the family. Well that is all for now, take care and keep collecting!
Currently myVidster's growth is at 16% (Source: Google Analytics). While this is great, I have to activity monitor the site to ensure that performance stays optimal. Most pages like the homepage, video bookmark and user profile will load within a second even during peak traffic. Performance starts to take a hit when viewing a collection homepage and user subscription page when the video bookmark count are in the 1k+ range. In this case performance is reduce to 2-4 seconds.
In response I have spent the last few weeks combing though my code and doing some house keeping cleaning up my code and updating sql queries. In the end I was only able to increase performance by 500-1000ms. On the bright side I did reduce the memory consumption of my PHP processes :).
Now we enter the Law of Diminishing Returns. Do I spend more time trying to optimize my code or do I upgrade or add new hardware? I know I can only do so much optimization before I get to the point that the benefits no longer justify the time I am spending (time is money). You also have side affects where an update intended for optimization breaks or limits site's functionality.
With that said, I will most likely add a new server to handle myVidster's growth. I being a geek always enjoy adding a new server to the family. Well that is all for now, take care and keep collecting!
Saturday, January 21, 2012
New Feature: Subscription Viewer
Yo dawg, I herd you like viewing your video subscriptions, so I put an subscription viewer in your chat bar so you can view your subscriptions while you watch videos.
I hope I got that meme right :). As stated in the meme (or lack of), you can now view your subscriptions while you navigate myVidster! Before I started developing this feature I had set 3 goals. The viewer needed to have a simple user interface, fast loading and non obtrusive. Here is a summary of how I accomplished these goals.
User Interface:
The viewer looks a lot like the existing subscription page, but with fewer features (you cannot search/display videos by user). Users are given the option to naviagate though their subscriptions by clicking the prev and next buttons. The state of the viewer (what page its on) is maintain while the user navigates the site. To notify the user of new videos collected by those they follow, the existing video bookmarks in the viewer will fade out and the new video bookmarks will fade in.
Fast Loading:
Smart caching is used to speed up load times and when the data needs refreshing the data is pulled from mysql memory tables. Other features like minify html/js/css are also used to minimize the 'payload' sent to the user's web browser.
Non Obtrusive:
By default the viewer will remain open and auto refresh is enabled. While I feel these features are great I know some users would prefer not to use them. I can see this being the case with users with low resolution monitors (especially older laptops) where the viewer is taking up too much space. With that in mind, users can disable the keep open and auto refresh feature.
I hope you will enjoy this new feature as much as I enjoyed coding it :). That is all for now and keep collecting!
User Interface:
The viewer looks a lot like the existing subscription page, but with fewer features (you cannot search/display videos by user). Users are given the option to naviagate though their subscriptions by clicking the prev and next buttons. The state of the viewer (what page its on) is maintain while the user navigates the site. To notify the user of new videos collected by those they follow, the existing video bookmarks in the viewer will fade out and the new video bookmarks will fade in.
Fast Loading:
Smart caching is used to speed up load times and when the data needs refreshing the data is pulled from mysql memory tables. Other features like minify html/js/css are also used to minimize the 'payload' sent to the user's web browser.
Non Obtrusive:
By default the viewer will remain open and auto refresh is enabled. While I feel these features are great I know some users would prefer not to use them. I can see this being the case with users with low resolution monitors (especially older laptops) where the viewer is taking up too much space. With that in mind, users can disable the keep open and auto refresh feature.
I hope you will enjoy this new feature as much as I enjoyed coding it :). That is all for now and keep collecting!
Sunday, January 1, 2012
A geeky explanation on myVidster's growth
It's time to start the new year by geeking out! December was a great month in terms of growth. To give you an idea of myVidster's growth rate I ran a few queries for video bookmarking.
Video bookmarks for the last 3 months.
Oct: 300,460
Nov: 308,709
Dec: 357,149
Now lets take the daily average for December which gives us 11,520 video bookmarks a day! We also have over 163,000 registered users.
To manage our growing community and to keep the site running smoothy I have added another server to the family :). Currently myVidster operates on 4 servers, now lets break down what each server is doing.
Server1: Web Server
Server2: Database
Server3: Chat/Messaging
Server4: Search
The servers do other important tasks, but the ones listed above are their primary tasks. I have also been hard at work with code optimization for the search, chat/messaging and database services. I will like a make a side note that chat/messaging performance has been improve greatly, so please message away!
Well that is it for now, take care and keep collecting!
Video bookmarks for the last 3 months.
Oct: 300,460
Nov: 308,709
Dec: 357,149
Now lets take the daily average for December which gives us 11,520 video bookmarks a day! We also have over 163,000 registered users.
To manage our growing community and to keep the site running smoothy I have added another server to the family :). Currently myVidster operates on 4 servers, now lets break down what each server is doing.
Server1: Web Server
Server2: Database
Server3: Chat/Messaging
Server4: Search
The servers do other important tasks, but the ones listed above are their primary tasks. I have also been hard at work with code optimization for the search, chat/messaging and database services. I will like a make a side note that chat/messaging performance has been improve greatly, so please message away!
Well that is it for now, take care and keep collecting!
Tuesday, November 15, 2011
New updates: sorting search results and power user recognition
Greetings Vidsters! In my ongoing efforts to make myVidster your first and primary destination for viewing and sharing video content I have pushed out two updates based on your feedback.
Sorting search results by most recent.
This feature was requested by a user a few weeks ago and I was 100% on board with adding the sorting option. There was one small issue, the search index was storing the post date as text and not as a date field. To fix this I had to rebuild the entire index which contains over 3 million documents. It took my server 3 days to complete the update. I also took the opportunity to make some minor styling updates to the search page, lets see if any of my power users can determine what changes I made ;)
Power User Recognition!
The key to the success of any social network/sharing site is its power users. They are activity engaged on the site by providing content (ie bookmarking videos), commenting, tweeting, inviting their friends to join and follow them, etc. With that said I wanted to give them some recognition. On myVidster's homepage I have replaced the Top User Collections section with Power User Collections. To be highlighted as a power user you need to link or upload a profile photo and are activity collecting videos. This list updates every 15 minutes with randomly selected users that meet the required conditions.
Please continue providing me with your feedback and keep collecting!
Sorting search results by most recent.
This feature was requested by a user a few weeks ago and I was 100% on board with adding the sorting option. There was one small issue, the search index was storing the post date as text and not as a date field. To fix this I had to rebuild the entire index which contains over 3 million documents. It took my server 3 days to complete the update. I also took the opportunity to make some minor styling updates to the search page, lets see if any of my power users can determine what changes I made ;)
Power User Recognition!
The key to the success of any social network/sharing site is its power users. They are activity engaged on the site by providing content (ie bookmarking videos), commenting, tweeting, inviting their friends to join and follow them, etc. With that said I wanted to give them some recognition. On myVidster's homepage I have replaced the Top User Collections section with Power User Collections. To be highlighted as a power user you need to link or upload a profile photo and are activity collecting videos. This list updates every 15 minutes with randomly selected users that meet the required conditions.
Please continue providing me with your feedback and keep collecting!
Subscribe to:
Posts (Atom)