SQL for Data Analytics · A mistake Claude made

A mistake Claude made

What the AI told me

This query on games, the video game sales table, counts the releases and adds up the copies sold for each year from 2008 on. Run it.

Releases and total copies sold for each year from 2008 on.
⌘/Ctrl + Enter

Taken at face value, the result says video game sales have fallen almost every year since 2009. The video game industry kept growing through those years, so something about the games table is off.

I work on this course with an AI assistant called Claude. While I was building Unit 04, I asked it to look through the games table for problems students could learn from. One of the things it reported was this.

2015 has 614 games and 264M copies. 2016 has 344 games and 71M. 2017 has 3 games, 2020 has 1. The data was collected partway through 2016, so the drop isn't real.

Compare Claude's report with the result of the by-year query. What does it leave out?

How I'd answer it

Every drop before 2016. Claude was right that the data was collected partway through 2016, so that year is only partly there. But sales fall almost every year from 2009 on, and the partial year explains none of the drop from 2009 to 2015. Claude explained the last drop and said "the drop isn't real" as if that covered all of it. It also listed the rows from 2017 and 2020 without noticing that a dataset collected in 2016 shouldn't have them, which Unit 04 pointed out.

When I pointed out that the drop started years before 2016, Claude said this about Unit 04, the unit we were building.

I went straight to the explanation I had for 2016 and never read the result as a whole. That's the same mistake the unit is about.

Claude had an explanation that was true and fit the biggest drop, so it stopped looking. When you have an explanation for a result, whether it came from an AI or from you, check that it fits the whole result before you accept it.

Claude is among the best AI models available, and it made this mistake while helping me build the lesson about it.

Pause and answer

These are just for you. They aren't graded or turned in.

You ask an AI how Northwind's sales changed from 1996 to 1997. It runs a correct query and tells you sales fell by almost two thirds. What would you look at before you passed that on?

How I'd answer it

How much of each year is in the Orders table. Northwind's orders run from July 4, 1996, to February 12, 1997. So 1996 has about six months of orders, and 1997 has about six weeks. Comparing the two years says nothing about whether sales went down. Broken down by month, January 1997 has more orders than any other month.