• It was brought up on our work Slack yesterday, and having the day off but stuck at home whilst the roof was being put on our new porch, I spent a couple of hours playing with it.
  • I made numerous efforts to get it to produce code I could personally use, and not once did it produce anything that was remotely usable, at least not with more work than creating it from scratch myself. 
  • At best, it produced examples of the sort I once provided in university classes to give a good idea of what an assignment required without giving away the real solution.
  • Where it might have use is generating regular expressions. I asked for several standard examples plus some custom problems, and it generated results. But I'm not sure they were better, or that it wasn't more work, than using a conventional regex generation tool.
  • I'm also not sure I'd trust it to be correct. I asked it to give me the last digit of pi. It confidently, though with some reasonable caveats, claimed it was 9, then gave me an example of pi with a dozen or so digits to demonstrate this fact. They ended in a 6.
  • I asked it for several fictional stories on given topics, then several interviews with famous historical figures. The results were all notionally competently written — the grammar was correct and they were easy to read — but dull and superficial, like those written by a bored high school student or a clickbait content writer who knew what the topics were about or what the historical figure did, but didn't care.
  • I asked it for instructions for starting a vintage motorcycle. The response was accurate, mostly, except for entirely leaving out the important bits that would avoid the risks of injury from misusing the kick-starter.
  • A colleague did a similar request for instructions on how to land a glider. He said he thought it was trying to kill him.
  • Attempts at natural conversation seemed stilted, dull, and joyless. It felt like a bad dinner date with a slightly stern, humourless and judgemental person who didn't like me.
  • Speaking of dinner, it did surprisingly well at generating recipes. Requests for vegan stew with chicken were appropriately rejected as contradictory, but (say) a request for vegan stew with lentils and potatoes produced what looked like a recipe for a pleasant, though perhaps somewhat uninspired dish. I suspect it would be a bit bland.
  • Though I'd happily use it for that. My overall impression was that it's what we should expect from a modern-era ELIZA, but like ELIZA it's still basically a toy, though it might make a great core for something like Google Assistant or Amazon Alexa.
But replacing programmers?
  • No, not by a long shot. When it can interview users and stakeholders and design a solution in light of company direction and corporate vision, whilst considering budget, timescale, available and preferred infrastructure, industry regulations and current technology trends and meet functional and non-functional requirements, then I'll be concerned.
  • For now, it doesn't even look like it could be a useful programmer's assistant; a capable mechanical co-worker that can quickly hammer out the repetitive dull parts so I can focus on the good bits. I think we'll have to wait for GPT4, or GPT5, or later, before we get that.

Post a Comment