What is big data & what classifies as big data?

Question

1.00/5 (2 votes)

See more:

I have went through a lot of articles but I dont seem to get a perfectly clear answer on what exactly a BIG DATA is. In one page I saw "any data which is bigger for your usage, is big data i.e. 100 MB is considered big data for your mailbox but not your hard disc". Whereas another article said "big data to be usually more than 1 TB with different volume / variety / velocity and couldn't be stored in a single system". Also that data should be stored in a NOSQL db with Hadoop used to transform data.

Further, I have been working on a solution and was wondering if I could classify it as a big data. Snippets on the solution below,

- Millions of raw data records and usually 500 plus GB of data.
- SQL database as back-end and SSIS / SQL queries to cleanse/process the data and convert it to a meaningful form.
- Visualization using Spotfire

Any help would be much appreciated. Thank you

What I have tried:

I have performed self research on several web sites like Quora, Stackoverflow and other personal blogs

Posted 22-Feb-16 7:29am

Chendur Srinivasan

Updated 1-Apr-16 23:41pm

Add a Solution

Comments

RedDk 22-Feb-16 13:55pm

From the standpoint of actually USING "BIG DATA" (as you've suggested such a thing exists) I'd have to say let it concern some platform which gives a developer access to it. In other words, let's say you have an SQL Server database and you're usg TSQL in procedures to get at the data. Unless\until there's a datatype called "BIG DATA" don't waste your time.

Maciej Los 22-Feb-16 15:33pm

This is my scale of values for data size:
small data = less then 10MB
medium data = less then 100MB
large data = less then 1TB
big data = more then 1TB
;)

Sascha Lefèvre 22-Feb-16 16:09pm

More than anything else "big data" is a "big hype train" for something that isn't new at all but apparently all new for the media.

2 solutions

Solution 1

How long is a piece of string?

The definition of "big" depends on you and your point of view, but the general consensus is what you said the 3 V's Volume/Variety/Velocity.

Also "big data" is considered what you store "as is" and process or compute answers later hence solutions like map/reduce and hadoop etc.

Posted 22-Feb-16 9:36am

Mehdi Gholam

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Maciej Los · Accepted Answer · 2016-02-22T09:52:00

I'd suggest to read this: Big data - Wikipedia, the free encyclopedia[^]

Wiki wrote:
What is considered "big data" varies depending on the capabilities of the users and their tools, and expanding capabilities make big data a moving target. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."

As you can see, there's no chance to define the portion of data which is identified as "Big data", because it depends on:
1) who is judging (person, organization/concern)
2) what technology is used for storage data (text file (txt, xml, csv), media files (such as mp3, video), etc.)
3) what devices is used (mobile device, local computer),
4) what technology is used to connect devices (Lan/Wan/WiFi)
5) etc.

My personal scale you can find in the comment to the question. Do not tie yourself to this scale, because as many people you'll ask as many answers you'll get.