Collecting Data for AI

To understand how AI works today, I find the best place to start is with data.  Every AI company out there, whether OpenAI, Anthropic, xAI or whoever, needs data.  It's the cornerstone of training AI models, which I will get into later.

Stage 1 - Where does the data come from?

This is where many people start their opposition to AI.  The way in which they harvest data to train their models.  You will hear comments like theft pretty often. Why?  Let's look at where the data comes from.

Public web data

This is the backbone of data collection.  It can be collected on a large scale using what's called web scraping, which is just a gritty term for helping yourself to anything on a public website.

• Crawling websites and simply taking the information contained in them
• Wikipedia
• News articles
• Blogs and forums
• Digital books
• Other public locations like GitHub

Licensed data

Sometimes web scraping results in copyright and legal issues, so where that is likely, AI companies will pay to access certain data.

• News Archives
• Book catalogues
• Academic journals
• Proprietary data from publishers

User-generated content

Only content that is publicly accessible on the web is actually used here.  AI can't automatically access private messages or private account content, but if it's given access, it can.

• Social media posts
• Forum discussions
• Q&A sites
• Code repositories

You get the idea; there is more, but basically, the entire accessible internet is the largest, most easily harvested source of data.  And boy do the AI companies harvest it well!

But let's spend a minute on all that public data: who owns it?

Data Ownership

If you have a website, this blog, for example or my other data site at propertyresearch. Both are mine; I produce all the content, so they legally belong to me, right?  Mmmm, kind of.  I will distil it;

If your site is publically accessible, anyone can technically fetch it.  But, nobody has the legal ight to use your content beyond what copyright allows, you can block any crawler at any time.

That said.  On this blog right here and now, I own the content; copyright is automatic.  Every post is protected the moment I create it.  That includes images if I created them (I don't, thanks Unsplash).

If someone logs in to this site and makes a comment on a post, that person owns the copyright to their own comment; however, I retain the right to display it.

Implied permission

Here is where it gets grey!  Search Engines like Google and Bing, as well as AI companies, operate based on something close to implied permission to crawl. If a site is publicly accessible and doesnt tell them to go away, they crawl it.

Withdrawing permission

As a site owner, my control is absolute.  Lol, just kidding. If you believe you control your data online, I have a new product to sell you: I call it negative temperature water. 

I can withdraw consent to anyone by blocking them.  Now, that is where it gets greyer and trickier.

If I wanted to stop Google, Bing, ChatGPT, Claude and others from crawling my website, I can.  It's easy if you know how.  I can simply add some code to a file called robots.txt that is on my website telling Google, " Not here, thanks.

The issue with that is that whilst I can tell all those bots not to scrape my website, I still can't stop them.  

That said, Google, Microsoft Bing, OpenAI, Anthropic, xAI and most others will respect my robots.txt file, see that they are being asked not to scrape or index the site and go away.

But there are lots of bad actors out there who will send bots to a website, see that the robots.txt file says "go away," ignore it, and continue.  To be clear, robots.txt is not security; it's signalling, asking bots to respect the site's wishes, not compelling or forcing them to.

Now I am not saying you should block everyone; getting listed in Google, Bing, or other search engines is actually what you want.  They are a helper in getting your site noticed and attracting more visitors.

You can tell good actors to stop and they will.  You cannot stop bad actors with robots.txt alone.

For the most part, you can withdraw consent, and most will respect that. Those that don't can be blocked by other means; I will come on to that.

Bad AI actors can cripple a website

I speak from experience on this one. Over at my propertyresearch site, I found this out a couple of months ago.  On that site, I have massive amounts of data, to me at least.  The database is over 150GB; if you consider how many rows and cells it contains, it's over one billion.  

I have 31m property records, 30m Energy Performance certificates, Property sales by postcode, street, and so much more.  80m+ different pages could be generated if someone visited every single one, but I had no concerns; nobody was going to do that, right?  True, no person tried to do that, but AI did!

One day, I started getting messages on LinkedIn from a few people saying the site was very slow.  Odd, I thought.  This is a powerful, dedicated server; it has an 8-core, 16-thread CPU, 64 GB of RAM, and so on.  Shouldnt even break a sweat!

When I logged into the server, I just saw red- not from me; well, actually, yes, me too- but it was the system telling me the entire CPU was running at 100% across every core. Can't be right; to be fair, it can do that if I forget to re-enable something after an update, but it wasn't that. It was a bad-acting AI bot.  It was crawling through my entire site, trying to scrape all of those 80m+ pages and, in the process, killing my server, and nobody could access the site.

I used AI that used massive amounts of public data in it's training to stop AI trying to crawl my website possibly for it's own training.

Yeah, to figure out what was happening and how to prevent it, I turned to ChatGPT, who in turn started looking into it, figured it out, and, between us, blocked the bad actor.  Within minutes, my CPU stopped cooking and went from 100% down to about 14%.  Phew, fixed.

So yes, I used AI to stop AI.  A bit of the hypocrite in me coming through again, huh.  I am not hiding it.

Let me give the 'good' actors a wave, though; OpenAI and others were also doing it. I never stopped them.  But they didnt hammer my CPU and bring my site to its knees!

So, my point is, you can stop the 'good' actors; you will have to fight with the bad actors, they don't care, they just want data and data and data and will do whatever they want to get it.

Right, rant over, so what happens next?

Stage 2 - Cleaning the data

Ok, so the AI company has all the data, billions of pages of data from many sources.  Great, so just feed it into something, like a neural network and start training it using all that data?  No, not yet. 

The thing is, there is lots of great data on the internet; we all know this, but it's also a massive garbage tip as well.  Not all of that harvested data is true, accurate or makes sense.  So it needs to be cleaned first.

• I finish this post and publish it; someone copies it, then 6 more, so 8 copies of the same thing
• Adverts all over the pages
• SEO junk
• Hate text
• pornography
• Personal data - Names, emails, phone numbers and so on
• Lots of endless junk that we all find and sigh about wasting our time searching for something

Those billions of pages need to be sorted into useful, not useful, useful in part and so on.

Its kind of like a giant laundry process, more complex than these three points, but you get the idea.

1. Clean it
2. Filter it
3. Deduplicate it

You know, from looking at websites, there are things like navigation menus, footers, boilerplate text; none of that is data, so it needs to be removed.  Somewhat ironic, an issue that AI has when harvesting data is that it comes across a lot of AI-generated junk; imagine that, huh!  It needs to go to: why would AI want its own crap?  And let's talk about that sometime; I have made a note: AI training that scrapes and uses its own data, or data from another AI, in its training!

Each company does it its own way; not going to get into the detail of that.  It's part of the process.

So, just to make up some numbers, you may have 3 billion pages of data after the cleanup, down from 5 billion.  

Now you just feed those billions of web pages, documents and other data straight into a model that will answer questions, right?  And, no!  It was never going to be that simple. 🙃

Next comes tokenisation.  Love that word.  Let's pick that up in the next article.