I had a spare half an hour this morning, and my newspaper had a list of 15 things the next generation of ChatGPT will be able to do for you. There's been some discussion about what ChatGPT does on discussion groups and how convincing it is. Someone, for example, asked it about clipping counters in Advanced Squad Leader and it gave reasonable arguments both in favour and against the idea. There's suggestions that it is just a really smart search engine with a text generator attached.
I thought it would be interesting to see if it could produce anything remotely serviceable for the historical notes for my upcoming set of War of Spanish Succession rules, so I asked it to describe infantry tactics during the War of Spanish Succession.
It started off all right. It got the dates right, and then talked about linear firepower tactics. It then went on to talk about the use of columns and skirmishers. I replied that this sounded like Napoleonic warfare, and it agreed with me, and modified its response (Changed its mind, perhaps?) to conform more with accepted understanding. So, it appeared to be learning. Or is it like that bloke in the pub who knows everything, but changes his mind to agree with you when caught out?
When pushed on the differences between French and British tactics, it then backslid dramatically and started talking about columns and voltigeurs. On being challenged it accepted it was wrong, and gave me an answer that suggested the French liked bayonets ("that make a musket like a type of spear") and charging, and the British liked firing.
Where it got interesting was when I asked about Dutch tactics. It started off okay, with a discussion of linear tactics and firepower. Then it got weird, and asserted that the Dutch used native troops from their colonies, armed with spears and bows and arrows, to attack enemy supply routes. I asked it to clarify if this was in Europe, and it assured me that it was, and that they were deployed in both France and Germany. It said they were known as the "Black Javanese". I pressed my little AI friend further. When were they deployed, I asked. The response was this:
Now that is REALLY specific. It almost made me doubt myself. I took the bull by the horns, and went straight in with the "I think you are mistaken" approach. Sure enough, it finally backed down, and admitted that "it appears that there is little to no historical evidence that Javanese troops were deployed to Europe during the War of Spanish Succession".
It took some pressure to get it to admit it had made a mistake, but we got there in the end but only because I knew it was wrong and asked the right questions. Reading what it wrote was very plausible. I did some follow up on Austrian tactics and the use of battalion guns, which was mostly convincing. I've had problems tracking down the exact usage of battalion guns in the WSS, so I'm not sure if the answer is completely reliable. I consequently won't be using any of its "research".
I moved on and asked it about the date of the battle of Edgcote. It went for the 26th July 1469. We had an argument. It insisted that the 26th July 1469 was a Monday (it wasn't - it was a Wednesday). It wasn't until I rubbed its metaphorical nose in the Coventry Leet Book that it finally caved and accepted the 24th. Where ever it is getting its information, it isn't from Wikipedia.
Where it did get uncomfortable was around the use of "Edgecote Moor" as the name of the battle. It insisted the earliest usage was in George Baker's History of Northampton written in 1822. It isn't, so I pressed it further. It revised its view and said it was in Hall's Chronicle in the 16th century. No it isn't. He refers to it as "Heggecote". When I told it I thought the earliest usage was in 1848, it agreed, and said that Baker's book was actually written in 1848. No it wasn't. And so it goes on. Like the pub bore, caught out in a lie, it changes its story repeatedly, mixing its facts up, and apparently making stuff up. It even quoted from a book it said was written in 1995, quoting title and author. The book doesn't exist.
So we have a machine that writes plausibly using an algorithm that seems to make stuff up, whether by design or accident. I sure won't be using it to submit any of my homework assignments.
On the other hand, it is making a decent fist of writing a set of wargame rules:
We are currently looking at each phase in turn. I have narrowed the parameters to 15mm figures on a 6' x 4' table, using 15mm figures. So far we have movement rates, and are working on the combat rules.
Watch this space. I will be making the final set available for download.
This is the second blog I have seen referring to using ChatGPT to write rules and interestingly (well, not very) at a recent work conference, our GM of Sales said he got it to write a business proposal for an imaginary customer in about ten minutes, and it did a better job than he would have done himself, after over thirty years hands on experience in our industry....
ReplyDeleteObviously, from your research, it has its limitations....goodness knows where it got the story about the Dutch colonial troops...sounds a bit like the WWI legend of Russian soldiers with snow on their boots being seen getting onto a train in Edinburgh!
If there is enough generic stuff out there to draw upon it seems it'll do a decent enough job. Focussing it on real detail will be an issue - look at the generalities in the outline rules. Nothing there to disagree on, but also nothing there you could use immediately.
DeleteAs to the Russian soldier story I always thought it was Sheffield. When I was at Uni there, a local new wave band was called "They Must Be Russians"
Amusing ... I was curious about the 26 July 1469 date, and googled it. Back came Monday (17 July 1469). Odd response, but there it was (thus confirming the answer to my question to be Wednesday.
ReplyDeleteBut I was wondering whether the Calendar made a difference. So I probed further. 26 July 1469 was Wednesday under the old Julian Calendar. What would that day have been under the Gregorian Calendar (supposing it had been adopted that early)? Monday.
At the time the Gregorian Calendar was adopted, the plan involved a 10-day shift, Thursday 4th October 1582 to be followed by Friday 15th October. The nine day difference in the 1469 dates (17 and 26) MIGHT suggest a nine day shift would have been appropriate back then. Maybe.
Interesting.
Cheers,
Ion
I'm confused here. What did you google to get the 17th July 1469? Hopefully not the battle of Edgcote! We're only interested in the Julian calendar for determining the date of the battle. Despite its introduction in the late 16th century in Europe, we were still using the Julian calendar until the mid 18th century in Britain.
DeleteNow, this is getting interesting!
DeleteGraham, when I read your book and your discussion of dates, my initial thought centered upon the date of the Feast of St. James. I wondered upon which date the Feast was celebrated in 1469? Was it 25 July (Gregorian) or the more logical 17 July Julian date? On the which date (Julian) was the Feast of St. James celebrated in 1469?
I punched 26 July 1469 (Gregorian) into a calendar converter, and like Ion, it returned 17 July (Julian), a Monday! Monday corroborates the day of the week cited in the Welsh poem.
If you give the date of battle as 24 July Julian, how will anyone in today's society (and calendar) know when to celebrate this battle? You now have me questioning the dates of all battles before the introduction of the Gregorian calendar!
The dates are all Julian. The battle can't have been fought on the 17th July (Julian), as all of Warwick's forces are still in Kent on that day. It is all a bit confusing as the calendars diverge by more and more as time goes on. As the Gregorian calendar wasn't introduced anywhere until over a hundred years after, looking at the primary sources and getting confused over what day they meant is a recipe for madness. It might not be an exact anniversary, but it has been hard enough getting people to move to accepting the 24th July, let alone then telling them actually the anniversary falls on the 17th. Mind you, if we shifted the anniversary of the 1460 Battle of Northampton back a week to the 3rd July we wouldn't clash with the Tewekesbury medieval festival anymore.
DeleteIf all dates are Julian then I am really puzzled! Today, the Feast of St. James is shown as being celebrated on 25 July. That is Gregorian, I suppose. Was the Feast of St. James celebrated on 25 July under the Julian calendar as well? Can't wrap my head around that...
DeleteThe Feast of St James is the 25th July. It was the 25th July when it was first thought of (probably sometime between 600 & 900 AD), so it is the 25th July in the Gregorian calendar, as the Julian calendar moved the date away from the actual 25th July. Think of it like this: Christmas Day is the 25th December, regardless of Julian or Gregorian calendar.
DeleteMy curiosity was aroused by the response by the ChatCPT to the question 'what day of the week was 26 July 1469. The thought crossed my mind that is was extrapolating the Gregorian Calendar back to that date to get the reply 'Wednesday'. So I asked of Google (more or less) what would that date have been had the Gregorian been in effect at that date. Wednesday was the response (though I'm not sure I was 100% convinced).
DeleteWhat makes things really confusing, though, is that the Gregorian Calendar was adopted at various times by various countries and regions anywhere between 1582 and 1923!
Cheers,
Ion
Yes. It is confusing! Normally I'd say we are interested in what day of the week people thought it was at the time. Phrasing the question with ChatGPT is important, and knowing whether it has got it right - or have a vague idea of what the right answer should be - is important. I have tried another experiment on a different subject in an attempt to use it as a super search engine. It really isn't reliable in that role.
DeleteVery interesting. Sounds like it takes the first thing it finds and talks about that. If you challenge it it then looks at the next thing.
ReplyDeleteScary that it seems very convincing in the first instance, which would make its subsequent "facts" more believable if you have no knowledge yourself.
It certainly takes more time "thinking" about some answers, so it may be weighing the probabilities when it finds more than one outcome. Rather than going with the first thing found it may look for several and then take the consensus. When you challenge it, it looks like it tries to find any evidence supporting your challenge, and then re-evaluates. This causes longer pauses. What is very "human" is its barefaced assertion of things that aren't true!
DeleteNice. I've been experimenting with AI-driven gaming and RPGs after reading some interesting blogposts about devising both units and figure models. Hadn't thought about rules at all.
ReplyDeleteFrom what I've seen so far, it is more useful for inspiration than immediate use, but inspiration is still quite a good thing.
It is useful for gathering ideas. Currently would work best for generic fantasy as its fact checking is very poor.
Delete