Is ChatGPT telling us the "truth"?

The effect of training sets on model outputs

For the first article after summer break, I’d like to expand upon a topic covered in my first post - how the data used to train AI models such as ChatGPT and DALL-E affects the outputs of those models. I recommend re-reading that article before continuing here if you’re new to the idea of AI. Many people think that typing a question into ChatGPT means that it will tell you “the truth”, but this is not the case, as evidenced by the lawyers who cited an imaginary case in their AI-generated legal brief in a now-infamous case of AI hallucination. Meanwhile, the (mis)use of AI in education is on the rise, with students using it to blindly copy and paste homework answers, assuming that it is telling us the correct answer.

There are some topics, such as math, where there is a definitely correct answer, and while AI on its own cannot compute the answer, there are tricks chatbot providers can do to ensure the model is giving the right answer. However, there are plenty of topics where there is no one “correct” answer - think of all the different essays kids would turn in on Hamlet in tenth-grade English class. What one person thinks is the correct way to interpret a play may differ greatly from another’s, and can interpretations be valid. It’s in areas like these where the training data can have the greatest effect on what the model says when asked about such topics. These answers can in turn have large effects on us as individuals and society as a whole, and that is why it is important for the general public to understand this before using AI tools, for the government to set rules about what data can be used, and for AI companies to provide information on said data.

To illustrate the point, let’s go back to the example from my very first post - a neural network that attempts to distinguish between hot dogs and everything else in the world that is not a hot dog. This may sound like an easy enough task - just get some pictures of hot dogs and pictures of other things, then train the model. However, there are nuances that can greatly affect the output of our model.

First, let’s say we don’t feel like labelling our data ourselves, so we hire someone on Amazon Mechanical Turk to label the data for us. We tell them to assign a label of “1” to a picture of a hot dog and a label of “0” to anything else. Let’s say our labeller doesn’t know why we’re asking for these labels, and they get this picture:

Dachshunds can be known as “hot dog dogs”, so maybe that is what we meant? The labeller labels this with a “1”. Now, when we train our model, it thinks that dachshunds are hot dogs! As humans we can see that is absurd, but the machine only knows what we have told it via the labels, and doesn’t question it. This problem is known colloquially as “garbage in, garbage out” - our labels don’t make sense, so neither will our model outputs.

Now consider a case where we don’t have a lot of money, so we decide to create a training set ourselves. We get a pack of hots dogs, hot dog buns, and various condiments, assemble a variety of hot dogs, take some photos, and train our models. All is going well until this photo:

Where the model guesses “not a hot dog”. How could this be possible? Well, in our limited dataset, we used hot dog buns with every example of hot dogs, so now our model thinks “I should only guess that something is a hot dog if I see a bun and a hot dog”. This is a classic example of a lack of training diversity preventing our model from generalizing.

Now we’ve learned our lesson, and so we walk around our hometown of Chicago and take photos of everyone we see eating hot dogs, and train our model. It works great until someone sends in this picture:

and the model confidently states “Not a hot dog”. How can this be? As memorably captured in Season 1 of The Bear, Chicagoans famously reject the idea of ketchup on hot dogs. None of our training data contained photos of hot dogs with ketchup, and now the model doesn’t recognize it. Due to a lack of cultural awareness around the way New Yorkers eat hot dogs, we didn’t even know that our dataset was missing important example. Issues like this underscores the need for subject matter experts in AI model development.

Finally, let’s say we fix these issues and release our AI model to the world, where it goes viral for its incredible ability to tell when something is a hot dog. All is going great, until we are approached by Beyond Meat. They love our model, and want the model to label their Beyond Meat product as hot dogs. But is a Beyond Meat hot dog the same as a beef hot dog? There is no longer a clear answer. This is a dangerous issue, since the model trainers are making a decision about what the model sees as “true”, but that decision is not necessarily clear to the users.