Master ChatGPT Today or Risk Obsolescence Tomorrow

on

As a dev I tried ChatGPT for a week instead of search engines, official docs, and Stack Overflow. Here's what I learned.

 


Since ChatGPT is all the rage I thought I'd try it for couple weeks instead of my existing dev workflow of search engines, Stack Overflow, and reading docs.




Even as it exists today, in beta, it has substantially increased my productivity, and I guarantee it will for you.

In this article I'll break down six areas where ChatGPT can help increase your productivity as a developer. Of course, no tool is perfect and so I'll also list six areas where it fails, so you can avoid those pitfalls.

However, there's something I can't easily convey here: its potential. While this article can help you understand some use cases, until you try it, it'll be hard to appreciate what it might be like in a year, two, or five. Large Language Models (LLM's) like ChatGPT will be the de facto way of working very soon, and once you try it, you'll realize that if you aren't on board, you're going to be left behind.

And with that bold prediction out of the way I'll get started:

#1 Pro: Find Information, Fast

ChatGPT is amazing at searching through all of humanity's knowledge to find an answer to very specific tech questions in a human readable way. For instance:

  • In bicep if I specify a system assigned identity, how can I get back the app id of that identity?
  • How do I set the AccessToken in order to connect to an Azure SQL Database using a service principal?
  • In Azure pipelines, what hosted options are there when setting the vmImage setting for a pool?
  • How do you convert an object to json using System.Text.Json?
  • In Bicep what are the pro's and con's of using an import statement vs using a module?

ChatGPT answered all of the questions above perfectly. For instance:




It was able to surface the information I needed faster than I possibly could have with my existing workflow, and it always provided helpful code samples.

#2 Con: Over-Confidence

Do you have a friend or family member who answers questions with such conviction and precision that you can't help but believe them? Except that you've eventually learned that they're mostly full of crap? That's ChatGPT (today). It will dazzle you at the precision of the answers and the way it provides code samples that look runnable. Except they aren't. For instance:

  • In a windows azure app service when you set the WEBSITE_LOAD_CERTIFICATES app setting, in what folder does the app service store certificates?

ChatGPT provided a specific folder that looked like exactly what I was looking for. Except it was a complete lie.

The same as with my CoPilot suggestions: don't make the mistake of trusting AI results at all. In many cases ChatGPT provides obsolete information, or provides answers that look perfect, except they just aren't valid.

#3 Pro: Conversational Follow-Ups


When ChatGPT gives you a crappy answer or code that doesn't run, you can tell it that it was wrong, and why, and it can follow-up for a better answer. For instance:

  • There is no `group` parameter on the command `az role assignment create` did you maybe mean `--assignee`?
  • After executing that I got role "Directory Readers" doesn't exist. Why might that be?
  • Your example didn't work, I got "Missing expression after unary operator '--'.". In your multi-line example do there need to be quotes around each argument?

Each time ChatGPT corrected itself and helped get to a correct answer. For instance, to that last point ChatGPT responded:

  • I apologize for the confusion caused. You are correct that the arguments field in the PowerShell@2 task should be a single string, and the arguments field should not span multiple lines…

It then provided a valid example that was helpful. ChatGPT can even follow up with abstract concepts like theoretical reasoning about variable naming!

Incredible!

#4 Con: Stale, Unprioritized Sources


As the saying goes: bad data in bad data out. In this respect ChatGPT has some serious information shortcomings:

  • It fails to take authoritative sources into account (e.g. learn.microsoft.com will contain correct information about Azure content vs random discussion groups)
  • It also fails to take age of information into account (newer should be better)
  • Its model is over a year old, which is like 10 in dev years, so its information is often stale

While it tends to be correct more often than not, it's information on how to do Open ID Connect authentication for example was repeatedly and frustratingly out of date and I wasted a ton of time trying to get information out of ChatGPT that it just couldn't provide.

#5 Pro: Help Debugging


Chat GPT can help you troubleshoot issues too. For instance:

  • The command `.\build.ps1 --target=PublishDevopsArtifacts` appears to be freezing in DevOps. It produces the output "xyz", any idea why?

ChatGPT provided three possible reasons and then presented six extremely helpful debugging techniques such as logging to a file and increasing the verbosity level.

#6 Con: ML That Doesn't Learn


Sadly, when you point out areas where ChatGPT fails, it doesn't learn from its mistakes in subsequent sessions. Like an amnesiac, it forgets every single session. For someone who likes to give back to the community such as adding answers or clarifications to old questions in Stack Overflow this is a big disappointment.

#7 Pro: Writes Code


In many cases ChatGPT can write small functions to allow you to work at a higher level of abstraction. For instance:
  • Using Azure CLI can you please show me how to 1. create an Active Directory group called "DirectoryReadersGroup"; 2. assign the "Directory Readers" role to the group; then 3. Add a SQL Managed identity called "MyProj-dev-sql" to the group.

Or even more complicated tasks like:

  • Please provide me a DevOps multi-stage build pipeline. It should run on `windows-latest`. It should trigger on merge to the `main` branch. The first stage should run the command `.\build.ps1 --target=PublishDevopsArtifacts`. The second stage should download build artifacts to a downloads directory, and then it should run the command `.\build.ps1 --target=RunDbMigratorDevOpsArtifact --environment=$environmentName --migratorZip='$(Pipeline.Workspace)/download/Migrator/Migrator.zip'`

The results are sometimes immediately usable such as in the first case, or in other cases a guideline that you can follow as you hand write something more complicated, such as in the second case.

#8 Con: Writes Bad Code


ChatGPT can't run or compile code at the moment, so it can't verify that any code it provides to you is correct. If a future version had a dev environment it would be substantially more helpful.

However, like giving a toddler car keys, perhaps giving AI access to a development environment is a terrible idea. So, I guess just get used to bad AI code for the foreseeable future.

#9 Pro: Translate Paradigms and Languages


Sometimes the docs you're reading provide a sample in a different language or paradigm. ChatGPT to the rescue:

  • What is the equivalent of "Get-AzureADServicePrincipal -SearchString "Microsoft Graph";" in Azure CLI?

Very helpful.

#10 Pro: Simplify laborious tasks


Dev work has plenty of tasks that require a lot of typing. For instance:

  • Can you please create the C# records (or classes) that will allow me to deserialize the following json: `{ "student": [ { "id":"01", "name": "Tom", "lastname": "Price" }, { "id":"02", "name": "Nick", "lastname": "Thameson" } ] }`

Pretty nice. Unfortunately, ChatGPT produced non-idiomatic lowercase C# and records whose properties weren't in the constructor. Nonetheless I was impressed and appreciate the potential.

#11 Con: Doesn't list Sources


As a dev I find it frustrating that ChatGPT answers so confidently but doesn't give any indication how it knows what it knows. The good news is you can kinda get sources if you ask. For instance:

  • Please provide the NuGet link to the Azure Active Directory Authentication Library (ADAL) you just mentioned
  • Can you please provide a link to the official docs for the Azure SQL database resource in your example?
  • Can you please point me to the documentation for that `PowerShell@2` command?

With that type of query, you can quickly jump to the most up to date data about what you're attempting to accomplish.

Summary


After a week of using ChatGPT as my primary source of information, I have learned that it is sometimes, but not always the best first place to go. It is a powerful tool, but it doesn't entirely replace search engines or Stack Overflow just yet.



So, will AI eliminate the role of developers? Of course not. Until it can translate what users mean rather than what they say, navigate contradictory requirements, or grasp the bigger picture of a project, then it won't eliminate the role of software developers. Also, LLM's still need to solve some big problems like stale information, identification of authoritative sources, inability to validate code, and honestly just lack of ability to improve.

However, as in any industry with big change comes both opportunity and risk. Those individuals that fail to adapt may well get left behind. However, ChatGPT is also a fantastic opportunity to get ahead of the curve, and prudent developers will start mastering this new tool ASAP.

Full disclosure: While ChatGPT didn't write this post, I did use it to help flesh out some of the ideas. Also, the header image was created by DALL-E.

Comments

Post a Comment