Wednesday, May 30, 2012

myVidster Embraces NoSQL

If you are a hardcore web techie you have seen the word NoSQL toss around.  In a nutshell its an alternative database management system.  Most websites (including myVidster) are build with mySQL, a traditional relational database management system (RDBMS).  For the most part mySQL is one of the best database solutions out there and will meet most (if not all) of your data needs.

As myVidster user base grows so does its need for more and more data.  When your data gets into millions of stored rows and you are trying to insert 3000+ rows of data every 2 minutes, mySQL faults begin to show.

In my experience mySQL does everything well expect for inserting large amounts of data in a given batch job and your queries can take a performance hit when you are have to sort large amounts of user data.  Most developers will agree with my issue with inserts, but some will argue that proper use of indexes will solve your performance issues when sorting.  I beg to differ.

Lets say you want to sort all user notification by date and return the last 5.  You will put an index on both the user id and the post date.  If the users has 5000 notifications, mySQL will run a filesort on the post date index (all 5000).  Disk I/O starts to become an issue when you are auto refreshing user's notification every minute.  Now if mySQL provided sorted indexes, then fetching the last 5 rows by date would not require a filesort, but sadly mySQL does not provide this feature.

I knew I have reach the limits of what mySQL can do so I decided to look into MongoDB. MongoDB is a NoSQL document oriented storage database.  During my testing MongoDB could insert documents (there like rows in a table) at a rate of 1000+/sec.  That solved my first problem, what about querying large data sets?  MongoDB provides sorted indexes and as long as you define the sort order of your indexes correctly (ascending or descending) MongoDB will not need to perform an nscanned (its like mySQL filesort) on the user's data.  I am also using MongoDB for backing up and achieving data as well.

Am I planning on replacing mySQL with MongoDB?  For now my answer is no, MongoDB does not support table linking and the code rewrite would be massive and the performance gain might not be worth it.  I do see myself using MongoDB for new features that I have in the pipeline.

Well, I think that is enough geek talk for now, take care and keep collecting!


Jyemenai said...

...this post turned me on, lol.

Anna Smith said...

Embraces NoSQL.... LOL..
Madhyamam Epaper is the one of the most popular Malayam Language NEwspaper. Dinathanthi Epaper is the Tamil Daily NEwspaper. Searching for Tarun Bharat Nagpur Online.

Anna university said...

What you said is absolutely correct, thanks for sharing :) anna university results 2017

Haseeb Khan said...

this is very interesting information keep it up nice sharing... Windows 7 Loader Activator

downloadnetflixapk said...

Thanks for sharing. It's great

Saurav Arora said...

Thanks for your personal marvelous posting!

I quite enjoyed reading it, you happen to be a great author.

I will make sure to bookmark your blog and will often come back in the future.
I want to encourage that you continue your great posts , have a nice weekend!

JANA GANGA said...

Psychologist in Chennai
Psychologist Doctor in Chennai
Psychotherapy in Chennai
Life Coach in Chennai

crack download said...

Hi there to everybody, it’s my first visit of this blog; Wondershare Dr.Fone Crack
Hi there to everybody, it’s my first visit of this blog;
this webpage consists of remarkable and actually excellent material in favor of visitors.
this webpage consists of remarkable and actually excellent material in favor of visitors.

RIchard Single said...

great work Malwarebytes Anti-Malware 3.3.1

elizabith said...
This comment has been removed by the author.
360 Total Security 9 Key said...

It’s actually very complicated in this busy life to listen to news on Television, thus I
simply use internet for that reason, and obtain the most recent information. fl studio 12 crack
Freemake Video Downloader Key

Oliver Maurice said...
This comment has been removed by the author.
Imran Hashmi said...

Amazing some works here keep sharing with us.
Lucky Patcher
Freemake Video

maria hashmi said...

I like this blog for a sharing.Its a good blog.

maria hashmi said...

Awesome blog I really impressed.
Cyber Link Photodirector

Spotify Premium said...

Download Spotify Premium APK for android.

fifa world cup 2018 said...

fifa world cup 2018 Russia will be the quadrennial International Football Competition that will be going to be kicked off from 14th June and run till 15th July 2018 in Russia.

Latest Technoogy said...

Kohls Credit Card Login - Portable My Kohl's Charge. Manage your record ANYTIME, ANYWHERE! Check your change, pay your bill, and view late history in a rush. Snap here to sign in to My Kohl's Charge.

Champa Kali said...

KMFUSA is the full-form of Kia Motors Financial Service, KMFUSA is also known as the online login service for the customers of Kia Motors Finance that offers a big range of loans for all the auto finance along with providing leasing solution for their customer all over the world. KMFUSA Login

maria hashmi said...

Amazing sharing, keep it up.
Gridinsoft Anti-Malware

maria hashmi said...

Great post, thanks for sharing.
Total Uninstall
FL Studio
GridinSoft Anti-Malware

Jonna Richard said...
This comment has been removed by the author.
Michael Jones said...

All Assignment Help is a web portal where students get help in making assignments for all the subjects, with the help of our experts. You will get 100% plagiarism free assignment. Expert’s consultation is also available for students. If they have any query they can contact with our experts anytime.

kepmari said...

TNEB Latest Recruitment 2018
TNEB Latest Recruitment 2018
TNEB Latest Recruitment 2018
TNEB Latest Recruitment 2018
TNEB Latest Recruitment 2018
TANGEDCO Latest Recruitment 2018

dll files fixer crack said...

Thank you for providing information about it