AI-Powered 'HorseGPT' Fails to Predict This Year's Kentucky Derby Winner (decrypt.co) 40
In 2016, an online "swarm intelligence" platform generated a correct prediction for the Kentucky Derby — naming all four top finishers, in order. (But the next year their predictions weren't even close, with TechRepublic suggesting 2016's race had an unusual cluster of just a few top racehorses.)
So this year Decrypt.co tried crafting their own system "that can be called up when the next Kentucky Derby draws near. There are a variety of ways to enlist artificial intelligence in horse racing. You could process reams of data based on your own methodology, trust a third-party pre-trained model, or even build a bespoke solution from the ground up. We decided to build a GPT we named HorseGPT to crunch the numbers and make the picks for us...
We carefully curated prompts to instill HorseGPT with expertise in data science specific to horse racing: how weather affects times, the role of jockeys and riding styles, the importance of post positions, and so on. We then fed it a mix of research papers and blogs covering the theoretical aspects of wagering, and layered on practical knowledge: how to read racing forms, what the statistics mean, which factors are most predictive, expert betting strategies, and more. Finally, we gave HorseGPT a wealth of historical Kentucky Derby data, arming it with the raw information needed to put its freshly imparted skills to use.
We unleashed HorseGPT on official racing forms for this year's Derby. We asked HorseGPT to carefully analyze each race's form, identify the top contenders, and recommend wager types and strategies based on deep background knowledge derived from race statistics.
So how did it do? HorseGPT picked two horses to win — both of which failed to do so. (Sierra Leone did finish second — in a rare three-way photo finish. But Fierceness finished... 15th.) It also recommended the same two horses if you were trying to pick the top two finishers in the correct order — a losing bet, since, again, Fierceness finished 15th.
But even worse, HorseGPT recommended betting on Just a Touch to finish in either first or second place. When the race was over, that horse finished dead last. (And when asked to pick the top three finishers in correct order, HorseGPT stuck with its choices for the top two — which finished #2 and #15 — and, again, Just a Touch, who came in last.)
When Google Gemini was asked to pick the winner by The Athletic, it first chose Catching Freedom (who finished 4th). But it then gave an entirely different answer when asked to predict the winner "with an Italian accent."
"The winner of the Kentucky Derby will be... Just a Touch! Si, that's-a right, the underdog! There will be much-a celebrating in the piazzas, thatta-a I guarantee!"
Again, Just a Touch came in last.
Decrypt noticed the same thing. "Interestingly enough, our HorseGPT AI agent and the other out-of-the-box chatbots seemed to agree with each other," the site notes, adding that HorseGPT also seemed to agree "with many expert analysts cited by the official Kentucky Derby website."
But there was one glimmer of insight into the 20-horse race. When asked to choose the top four finishers in order, HorseGPT repeated those same losing picks — which finished #2, #15, and #20. But then it added two more underdogs for fourth place finishers, "based on their potential to outperform expectations under muddy conditions." One of those two horses — Domestic Product — finished in 13th place.
But the other of the two horses was Mystik Dan — who came in first.
Mystik Dan appeared in only one of the six "Top 10 Finishers" lists (created by humans) at the official Kentucky Derby site... in the #10 position.
So this year Decrypt.co tried crafting their own system "that can be called up when the next Kentucky Derby draws near. There are a variety of ways to enlist artificial intelligence in horse racing. You could process reams of data based on your own methodology, trust a third-party pre-trained model, or even build a bespoke solution from the ground up. We decided to build a GPT we named HorseGPT to crunch the numbers and make the picks for us...
We carefully curated prompts to instill HorseGPT with expertise in data science specific to horse racing: how weather affects times, the role of jockeys and riding styles, the importance of post positions, and so on. We then fed it a mix of research papers and blogs covering the theoretical aspects of wagering, and layered on practical knowledge: how to read racing forms, what the statistics mean, which factors are most predictive, expert betting strategies, and more. Finally, we gave HorseGPT a wealth of historical Kentucky Derby data, arming it with the raw information needed to put its freshly imparted skills to use.
We unleashed HorseGPT on official racing forms for this year's Derby. We asked HorseGPT to carefully analyze each race's form, identify the top contenders, and recommend wager types and strategies based on deep background knowledge derived from race statistics.
So how did it do? HorseGPT picked two horses to win — both of which failed to do so. (Sierra Leone did finish second — in a rare three-way photo finish. But Fierceness finished... 15th.) It also recommended the same two horses if you were trying to pick the top two finishers in the correct order — a losing bet, since, again, Fierceness finished 15th.
But even worse, HorseGPT recommended betting on Just a Touch to finish in either first or second place. When the race was over, that horse finished dead last. (And when asked to pick the top three finishers in correct order, HorseGPT stuck with its choices for the top two — which finished #2 and #15 — and, again, Just a Touch, who came in last.)
When Google Gemini was asked to pick the winner by The Athletic, it first chose Catching Freedom (who finished 4th). But it then gave an entirely different answer when asked to predict the winner "with an Italian accent."
"The winner of the Kentucky Derby will be... Just a Touch! Si, that's-a right, the underdog! There will be much-a celebrating in the piazzas, thatta-a I guarantee!"
Again, Just a Touch came in last.
Decrypt noticed the same thing. "Interestingly enough, our HorseGPT AI agent and the other out-of-the-box chatbots seemed to agree with each other," the site notes, adding that HorseGPT also seemed to agree "with many expert analysts cited by the official Kentucky Derby website."
But there was one glimmer of insight into the 20-horse race. When asked to choose the top four finishers in order, HorseGPT repeated those same losing picks — which finished #2, #15, and #20. But then it added two more underdogs for fourth place finishers, "based on their potential to outperform expectations under muddy conditions." One of those two horses — Domestic Product — finished in 13th place.
But the other of the two horses was Mystik Dan — who came in first.
Mystik Dan appeared in only one of the six "Top 10 Finishers" lists (created by humans) at the official Kentucky Derby site... in the #10 position.
they fixed the bet 60 system in later roms! (Score:3)
they fixed the bet 60 system in later roms!
no-brainer (Score:5, Funny)
The AI was trying to sucker people into betting on different horses than it did.
Re: (Score:1)
HorseshitGTP
This is just embarrassing (Score:2)
I haven't kept up with it does Madden football still predict the winner of the super bowl ev
Re: (Score:2)
This guy [bloomberg.com] became rather famous at it. On the one hand it's getting harder because the odds are more accurate and there's more smart analysis. On the other hand there are a lot of difficult variables to quantify, e.g. trip quality. The required nerves of steel are impressive and it's something you have to be in for the
Re:This is just embarrassing (Score:4, Insightful)
If it wasn't that hard, more people would be doing it. Of course if you understand how gambling actually works, a sure thing doesn't have very good returns. If there were a good statistical mode that worked more often than not that anyone could play, everyone would be doing it which makes the payout bad.
The only people for who gambling is profitable are the book keepers. They don't care who wins or loses because they get their cut either way. No game of chance has a positive return and the house will gladly let you think you have some kind of system to beat the odds. Anything that isn't a pure game of chance is gambler vs. gambler with the house taking their cut. They always win in the end, which is how they can afford to keep the lights on.
Re: (Score:2)
The only people for who gambling is profitable are the book keepers. They don't care who wins or loses because they get their cut either way.
No, bookmakers absolutely do care who wins or loses. Here is one example [foxsports.com] from a few months ago.
Zachary Lucas, director of retail sports at TwinSpires Sportsbook, knew pregame that Dallas was a serious liability issue.
"There's a landslide of money on Dallas. We're up to our neck in liability," Lucas said.
Fortunately, for them, Dallas lost so that liability went away. Had Dallas won all those bookmakers would have been paying through the nose.
For another example, look at what happened in this year's Super Bowl [morningstar.com]. Same game, two different bookmakers, two different results.
And fi
Re: (Score:1)
bookmakers absolutely do care who wins or loses. Here is one example [foxsports.com] from a few months ago.
Zachary Lucas, director of retail sports at TwinSpires Sportsbook, knew pregame that Dallas was a serious liability issue.
"There's a landslide of money on Dallas. We're up to our neck in liability," Lucas said.
One thing that has always been true - what bookies say about their results and what their results actually are are two quite different things. In the long run what happens in any particular event isn't important. The long-term profit is determined simply by how many bets they took in. The more business they write the more money they make. Sure, there can be outlier events, but they take steps to mitigate those so they don't go broke.
Re: (Score:2)
Bookmakers want a balanced book so that they pay out the same amount of money regardless of who wins.
Re: This is just embarrassing (Score:1)
Re: (Score:2)
A balanced book doesn't mean 50:50, it means that it is inversely proportional to the odds.
Re: (Score:1)
balanced book
Sorry, not great at reading minds.
Re: (Score:1)
This isn't really true. It's true that bookmakers do their best to set the odds so that they get about the same amount of money betting either way. In that case, they get their vig and do well no matter what the outcome. They don't always succeed, though, so there are cases where there's a lot more money on one outcome than the other and the bookmakers wind up havin
Re: (Score:3)
I knew a professional gambler. He made good money.
Sure. All professional gamblers make money. Just ask them.
But, for some reason, they're always broke.
One trick to manage their self-delusion is to retroactively put their winnings in the "professional gambling" pile and their losses in the "just playing for fun so it doesn't count" pile.
Re: (Score:2)
That's because they're gambling wrong. When you gamble in the stock market, the odds are in your favor, unlike in a casino.
Re: This is just embarrassing (Score:2)
The last part makes them think they're ahead, because they develop forgetfulness about that second pile "which doesn't really count" that is, until it includes this month's mortgage payment or a wedding ring
Re: (Score:1)
Sure. All professional gamblers make money. Just ask them.
But, for some reason, they're always broke.
One trick to manage their self-delusion is to retroactively put their winnings in the "professional gambling" pile and their losses in the "just playing for fun so it doesn't count" pile.
"Expenses" is also a handy excuse. "It wasn't the gambling - it was the expenses that ate up my bankroll." Followed of course by asking to borrow money.
Re: (Score:2)
"The hard part is it requires nerves of steel because you will be up 100,000 one day and down 200,000 the next day and up $120,000 a day after that. The point is it's all doable mathematically. "
That's what's wrong with this article. They evaluate the model on its accuracy, or lack thereof, on a single race. That doesn't work
Don't get me wrong, I wouldn't expect anything GPT-like to outperform a more simple statistical model on sports gambling. Still, this article doesn't make a lot of sense.
What a language
Re: This is just embarrassing (Score:1)
They'd be better off... (Score:2)
They'd be better off finding and using the next Mr. Ed. Or more practical, oopsy. JoshK.
Allusion... https://en.wikipedia.org/wiki/... [wikipedia.org]
LLM? (Score:3, Insightful)
You're telling me you tried a statistical predict with an LLM and it failed? Why not just use statistical prediction with traditional machine learning algorithms? Instead of assuming LLMs are magic?
Re: (Score:2)
Indeed. This is stupid. An LLM is totally the wrong tool for the job.
Even with a soft-max statistical model, the outcome of a single prediction means almost nothing. You need to look at the win/loss ratio over a thousand races.
Re: (Score:3)
But look at how /confident/ it is with the answers it gave! With that confidence you know it's a sure thing.
Re: (Score:2)
Because people have no idea how this shit works. Is this really news to you?
Re: (Score:2)
There was no "glimmer of insight" (Score:5, Insightful)
But there was one glimmer of insight into the 20-horse race. When asked to choose the top four finishers in order, HorseGPT repeated those same losing picks — which finished #2, #15, and #20. But then it added two more underdogs for fourth place finishers, "based on their potential to outperform expectations under muddy conditions." One of those two horses — Domestic Product — finished in 13th place. But the other of the two horses was Mystik Dan — who came in first.
This wasn't a glimmer of insight. It's completely irrelevant that one of the two choices it picked to come in 4th on a muddy track happened to win on a fast (dry) track. There was no possible betting strategy which would have earned a profit on these picks. And it would not have made these picks with the correct track condition.
Even if it had picked these 5 horses as potential winners on an accurate conditions, it still had a 25% chance of being right - making this a prediction about as reliable as "predicting" on a 4 answer multiple choice question.
HorseGPT was always going to be bad at its stated purpose, if had any value, they would have kept it private and made way more money betting the races than they could ever make selling the AI.
Re: (Score:2)
It's probably possible to do this with neural netw (Score:4, Interesting)
Re: (Score:2)
It definitely isn't the right architecture.
What you actually want to do is calculate probabilities of winning, and back horses where the odds given are better than that probability.
If your probabilities are more accurate than bookie odds, then, you will still lose most of your bets, but the ones you win will more than make up for it.
Baffling (Score:4, Insightful)
Why would anyone think an LLM would be able to predict the future?
Re: (Score:2)
Why would anyone think an LLM would be able to predict the future?
Exactly. The article even notes that it is mostly the same as the human predictions that it is ingesting. Why would anyone think it would have something different that what humans are predicting since that is what is learning from.
not even a trained network ? (Score:2)
wait so it's just a general purpose LLM given some reading material and asked to make predictions ? failure there is not very interesting. i'd be way more interested in a custom-trained network for horse betting.
You call this a summary? (Score:5, Insightful)
Oh for the love of all that is good, can Slashdot get some real editors? This is far too long to be a summary of the article, and there's no clear link back to the original story. Back in the CmdrTaco days, a Slashdot story consisted of a few sentences describing the main concept with a clear and obvious link back to the original story, It may have also had a key paragraph from the original story too.
Yes this is completely off topic but I'm so tired of the shitty editing. No wonder Slashdot has gone quiet.
ShitGPT (Score:1)
just Saw that AI in a bar (Score:2)
It was nursing an old fashioned...
Not a suprise (Score:2)
Not a surprise.
During the NCAA tournament I saw ads of someone saying their simulation/algorithm predicted around a decent number of the games of the first 2 rounds right. It sounded good, until I came up with a algorithm that easily beat their simulation badly (simply pick the lower seed).
Lots of algorithms that "appear" to work until you realize that with trivial simple algorithm(like pick the lower seed) that you can beat it with a lot less work.
My JockeyGPT did (Score:2)
nft
The Smart Money Stays in your pockets! (Score:1)