Sometime around Friday night, I was reading an article on Mashable about Weather Data and problems with social media, which I started and then abandoned a blog post on due to not feeling like I had a good angle (it’s a good article though and I recommend it). However, before I deleted the draft, I had been researching some data on how big the Internet is, and I found a fair amount of amazing information which I wanted to share in a post.
I assumed that the Internet is expanding at an exponential range, like of like the Universe really, and I was looking for articles to support this.
But according to World Wide Web Size, a site which tracks the number of indexed pages, the number of indexed pages doesn’t necessarily go up consistently and even goes down at times. This is because individual pages and whole sites are constantly going up and down. This gives the impression that the Internet isn’t expanding all that quickly despite the amount of content on it, but then you get this quote from Websitemagazine.com.
As of 2014 Google has indexed 200 Terabytes (TB) of data. To put that into perspective 1 TB is equivalent to 1024 Gigabytes (GB). However, Google’s 200 TB is just an estimated 0.004 percent of the total Internet.
That linked article above also has a genuinely fascinating infographic which tries to put in perspective how we use the internet as well as it’s sheer size in data.
This article also notes that a lot of the unindexed web is the Dark Web. So if you look at the galaxy image above, and imagine that each of those stars is a website, all of the space in between them is the Dark Web (sorry, that’s a rough analogy! It’s kind of the easiest way for me to understand it though!)