"Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions." ~Wikipedia
“Every two days, we create as much information as we did from the dawn of civilization up until 2003” ~ Eric Schmidt, former Google CEO
When I heard about this "Google doesn’t just connect us with content. It affects our perception of content" I was confused about that. If we consider the computer world progresses in five decades, nobody could imagine personal computers about 60 years ago and also reach to any data by internet about 20 years ago and etc. now is the era of petabytes of information and as computers of 40 years ago cannot handle information of 10 years ago, today's large and also smart data cannot be handled by last decade technologies too. The issue of Big data comes whenever traditional data management couldn't manage today's complex processes and a large amount of data.
"more than 85% of Fortune 500 organizations will fail to effectively exploit Big Data for competitive advantage." ~ link
Databases are in two types: analytical and transactional. Transactional databases are for structure databases and while analytical databases are for both structured and unstructured databases. Big data is difficult to work with using most relational database management systems, relational database management is good for a huge and powerful database server while Big Data is for massively parallel management on thousands of servers to high processing speed. Understanding big data needs to have a field of experience working in multiple database areas while it is especially for very complex business strategies and a simple application doesn't need any kind of big data to get used. So you have to have an experienced working with all type of structured and unstructured data to know what is going in big data.
as it's shown in the picture below, Big data is the mixture of any kind of data - structured (databases, sensor, clickstream, and location) and unstructured(text, data, email, HTML, social data and images, audio, video)
another issue is about migration to Big data, it seems there is no any good practice to migrate from because it comprised of a large pool of modern databases, platforms, software packages and tools, and migration to this kind of modern technology requires training and disbursement extend gigabytes of data to petabytes. There are some new tools like SSD, Hadoop and some technics like network virtualization to use. Extracting actionable intelligence from Big Data requires handling large amounts of disparate data and processing it very quickly. Considering scale and agility is one of the most important steps in migrating to big data, These issues must be considered when choosing your computing tools and your business processes. Maintain agility or flexibility, because the application gets under more and more changes to use big data, so agility and estimating the scope Is something valuable in big data. As skilled programmers and moderns tools do Not guarantee the success of a project, big data needs agility to achieve to be successful too.
I found these websites useful:
- http://en.wikipedia.org/wiki/Big_data
- http://www.sas.com/big-data/
- http://www.scaledb.com/big-data.php
- http://123socialmedia.com/7-big-data-articles-you-should-read/
- http://www.microsoft.com/en-us/sqlserver/solutions-technologies/business-intelligence/big-data.aspx
- http://www.mckinsey.com/insights/strategy/are_you_ready_for_the_era_of_big_data
- http://www.infoworld.com/d/cloud-computing/big-data-in-the-cloud-its-time-experiment-193152
- http://www.fsn.co.uk/channel_bi_bpm_cpm/mastering_big_data_cfo_strategies_to_transform_insight_into_opportunity#.UdsUS_mGpN9