Q&A: GitHub COO on how genAI makes devs more efficient (and can automate the helpdesk)

GitHub, the online developer platform that allows users to create, store, manage, and share their code, has been on a generative AI (genA) journey since before ChatGPT or Copilot was widely available to the public.

Through an early partnership with Microsoft, the dev platform adopted Copilot two-and-a-half years ago, tweaking it to create its own version — GitHub Copilot.

The genAI-baed conversational chat interface is now used as a tool for both GitHub users and internal employees to assist in code development, as well as an automated help desk tool.

There are people who believe as genAI continues to evolve and can produce more code based solely on user requests, developers will no longer be needed. As Nvidia CEO Jensen Huang last week said, because of AI, “everybody in the world is now a programmer. This is the miracle of artificial intelligence.”

Instead of software development, Huang believes humans should focus on more important skills such as biology, education, manufacturing, or farming, and the language of programming is now the human language.

GitHub

Kyle Daigle, who’s worked at GitHub for 11 years, took over as its COO about a year ago. He’s been part of a genAI development strategy that’s focused on discovering how the technology can benefit its roughly 3,000 employees — developers and non-developers alike — and its external developer community of users. So far, genAI is making developers 55% more productive, he said.

Daigle spoke to Computerworld about the various ways genAI has created efficiencies and helps both developers and non-developers. The following are excerpts from that interview:

When did you deploy Copilot. Why? And what has it enabled GitHub to do? “We’ve been on a journey with Copilot for about two and a half years now. We started working on Copilot when we got early access to the OpenAI models through our partnership with Microsoft. Similar to a lot of companies now, the main question was how do we put these LLMs to good use? It took us a little while to figure out the secret sauce that’s now Copilot. Originally, when we were using the models, we thought we were going to be building tools that documented code. You know, you give it your repository and it would spit out what that code did.

“But through experimentation, the idea of Ghost text — the sort of completion model of what Copilot does, where it shows you the entirety of a single message versus a single line — was kind of a big breakthrough for being able to get the most out of a powerful tool. And so, fast forward and now we’ve got more than a million GitHub users using Copilot every day. Our stats show it’s making them 55% more productive, and it’s writing about 60% of code; we expect that to get up to about 80% of code over time in many languages.

We do find it actually makes people more productive, happier and ultimately removes their toil.

“I think most importantly, and something we talk to our internal teams a lot about, is that it’s making developers feel more fulfilled. It’s allowing them to do more creative work and not the toil work. Instead of looking for the genAI to do the creative work, we’re allowing the human be in the pilot’s seat as the developer.

“So, we’ve had a lot of success during this time in getting every dev in our world allowing GitHub in helping them to write code.

“Internally, we’ve been focused on taking the lessons from Copilot and applying that to other places where we use AI tools, including Copilot outside of software development use cases. And of course other purpose-built tools as they come to market to help us all be a bit more productive.”

Assisting in code development seems to be one of the earliest low-hanging fruits your genAI platform addressed. How long have you been using it to assist in code production, and what languages was it helping with? “In our early experimentation, we were doing a lot of work in Python, JavaScript and languages like that. GitHub is mainly a Ruby company, but we also write in Go, and C, and FirGit. And so we were expanding our use cases of Copilot and using it in different languages. But overall, Copilot is able to work on the vast majority of languages that are in the public sphere.

“If you have a proprietary language, it can emulate that language because it’s looking at the code inside your repository and it can do a pretty good job of figuring out what it needs to use to feed the next line of code or next method.

“So, we’ve gone from a couple of test languages to essentially every modern programming language that has enough context in open source and the Internet.”

How is Copilot doing in terms of completing code? “In terms of code completion rates, what we’re talking about is in some cases when writing with Copilot, it might finish one line of code, but it might also finish an entire method; it might finish an entire file or class depending on the language you’re in.

“With something like Copilot Chat, you can talk to Copilot and say, ‘This is the problem I’m trying to solve,’ and it can potentially generate the entirety of a file for you. You can then say, ‘I don’t want it to be blue, I want it to be red. Or I want it to use this API or that API’ — those sort of adjustments.

“When we talk about the amount of code generated, we’re talking about the amount of code Copilot gave and the amount the user kept over time. Obviously, when you get a completion, you may think, ‘Oh, that’s not right.’ Or if you’re a developer you may have it write out a bunch of code and then realize I can refactor it right now; this isn’t quite right.

Sometimes the best code is no code at all and it’s just using human language.

“What we’re seeing is the vast majority of the code generated by Copilot is kept. Then down the line, after you write codes, you submit a PR, you run continuous integration, the next steps are also faster. So, code review on code that was written by Copilot with a developer tends to be faster because the code ends up being more correct. Continuous integration ends up being green more often than red on the first build because the code tends to be more correct. So there are a lot of interesting impacts  downstream, too, when you’re able to use Copilot as part of your workflow as a dev.”

Have you seen any issues — inaccuracies — while using Copilot? “The code that we’re providing you is also mimicking the code it sees in your repository. So in some instances, if it has an older code base, it’s going to take that into account and maybe do a practice that’s no longer modern, like the casing of variables or if you have a library that it’s going to call because it’s trying to emulate your existing project along with the underlying model. So for some developers, how they do that is use Copilot chat to say, ‘Hey, we’re actively upgrading this right now, so I’d like to use the new approach.’ And that’s one way we get through it.

“But also, we’re taking a lot of the technology we have with Copilot and the AI models we have underneath it and also finding other places we can help developers remain secure by fixing vulnerabilities as well. So, one of the things we shared at the GitHub Universe conference in November was something called Security Autofix, which uses similar underlying AI technology. So when tell it you are aware of a vulnerability…in a code base, we’re not just going to tell you you have a vulnerability you need to fix;  we’re also going to do the fix right there. All you have to do is say, ‘Yes, I’m good to go.’

“So, Copilot is still in the ‘copilot seat.’ You still need to follow best practices. You still need to do security scanning and secret scanning and all the things that have been true for good software development practices, regardless. But we’re trying to make ways to bring AI throughout the entirety of the software development lifecycle on GitHub to help you out. So it’s not just in your IDE when you’re writing code.”

Some in the tech industry are concerned that AI’s ability to automatically generate code could eliminate developers. What’s you take? “I think throughout modern history, there are so many moments when a piece of technology has come into the world, like the printing press, and everyone was like, hold on a minute —where’s my job going to go? But really what happened is a lot of work and opportunities that weren’t economical anymore were suddenly economical, because you’re not having your developers spend 60% ro 70% of their time solving problems that have been solved dozens, hundreds and thousands of times.

What we’re seeing is the vast majority of the code generated by Copilot is kept.

“So, the reality is…GitHub Copilot is making developers 55% more productive. Some customers have the reaction, ‘Does that mean I just get 55% of my developer’s time back?’ The reality is what your developers get back is the ability to solve harder problems more creatively than they previously had been able to when they had to do all that rote work themselves.

“So at GitHub we’re still hiring developers. We’re hiring right now. Really, what we find is we’re focusing more time on that upfront discussion or architecture and the problem we’re solving with customers. Because as the coding becomes faster, it becomes more important that you’re spending your time doing that creative problem solving work rather than that rote work that we’ve all done at some point in our career.

One of the major pieces of Copilot that goes under appreciated is the ability to do learning and development, or upskilling on the job.

“I’m more excited about the opportunity AI is starting to give us, which is we can do more than we did before because we’re able to write more code than we have previously. So, we can solve bigger problems that maybe before wasn’t possible, like that big re-write of your app. So many customers put that off and put it off because how would you ever be able to afford it. But, if it’s 50% less expensive, well maybe you can move to that new technology and you can use that new technology to move even faster in solving the next set of problems.

“I think we have plenty of work left to do. By every measure of every analyst in the world, there’s not enough developers in the world. So, we still have quite a gap to go before we have to worry about developers not having enough work to do.”

What benefits are non-developer teams seeing from genAI? “One of the major pieces of Copilot that goes under appreciated is the ability to do learning and development, or upskilling on the job. You have someone that’s new to a role, a company, a language and being able to come in and immediately have someone there you can ask questions of, that you can write code with and get that immediate feedback loop with Copilot.

“It’s not just the new folks. It’s also in a lot of cases those most senior developers we have where they get put in those nasty old projects that are extremely important to the business but have been just sitting on the shelves in the back closet somewhere keeping everything running. When they need to go into those projects and make updates, that L&D side has been really huge because they can go in and say, ‘I know Java, but I don’t know Scala. Or I know Java, but I don’t know .Net,’ or whatever. And, can you help me know what the next steps are?

“In a similar vein, we also focus Copilot on that user experience, i.e., you just start typing. There’s no real enablement; there’s no learning, there’s no buttons to figure out.

“So, I took those two learnings and we started to look internally into other places where we thought there was toil and where we could implement AI without having to enable AI. That’s the real secret. If you have to teach people how to use it, it’s not much better than any other technology choice you could make.

So, what were your other big wins? “One of the first big wins we had at GitHub, and I’ve seen this at other places, is in taking AI and bringing it into the IT setting. We’ve got a little over 3,000 Hubbers [GitHub employees] that have entered in hundreds and hundreds and hundreds of [helpdesk] tickets in an old system, to get assistance on why their laptop isn’t working, how they get access to the VPN, and so on.

“GitHub essentially runs in Slack. We’re a remote-first company. We’ve got employees all over the world; we’re not a return-to-office company. What we did was say, if we’re all on Slack, why not make that interaction with IT in Slack also be AI powered?

“So, instead of going into a portal and submitting a ticket, we have a channel called IT Help Desk, and in that channel is a bot that we call OctoBot. And when you ask a question, a vendor called MoveWorks that we partnered with … sees that question and OctoBot will come and say, ‘Yeah, I know exactly what you need to do. Here are your next steps.’ And in many cases we can even automate the workflow to say…, ‘we’ll  go set this up in these other systems for you.’

“Maybe it’s what every developer wants and it’s your laptop upgrade day and you’ve waited your two years. Click this bot and we’ll ship you your new laptop, and because we didn’t create a new system and didn’t have to teach anyone about a new portal, we’ve seen a huge improvement. We’ve gotten to where OctoBot is solving 30% of these tickets right out of the gate. We’re saving hours per day of every IT staff member’s time that we’re able to reinvest in other AI initiatives.

GitHub

“And, most interestingly, customer satisfaction…went up 12 points. So, now it’s at 98% customer satisfaction. I didn’t have to teach anyone. We didn’t have to roll out this technology in a way that you normally would in a corporate setting. We kind of just turned it on in a place where everyone was already asking for help.

“That’s been the big secret, I think, that when I talk to peers in other companies about how you can remove actual toil and save people’s time every day without introducing net-new behaviors? This is one example.

“Now we’re taking that process and testing about six to seven other tools currently inside the company. About 10% of our employee base is in some sort of AI tool trial, where we’re doing the same process: how do we do it without enablement? Can we put it in flow? Then we just measure the time saved on the other side.

“If it saves time and the tool pays for itself — onward and upward. What we don’t do is have a tool where you have to learn about it, get trained on it, or it’s very open ended. Like, this tool can do everything for you! Because we do find it actually makes people more productive, happier and ultimately removes their toil.”

We’re trying out other tools that are giving back employees potentially 30 minutes to an hour every day. Then they get to do the work that’s more strategic for our customers and move more quickly.

Are you considering other genAI tools? “In terms of GitHub…, we’ve taken a lot of the underlying Copilot technology and we’ve applied it to our support use cases. So, when you’re on GitHub and you have a problem and have to enter a ticket, it’s very similar to how we’ve learned from our IT experience. When you start to go through the flow, and we do the normal process of how we go through the problem, there’s a stage there where you can end up talking to Copilot. And we call it Support Copilot.

“It basically is able to walk it through with the context of what you’re asking and what access you have, potentially solving your problem right there.

“On the other side of it, looking externally, we’ve also been going past the idea that Copilot is helping you write code and helping you summarize a pull request.

“What happens when you sort of flip the model on its head and you start describing what problem you’re trying to solve to Copilot? That’s the exploration we’re working on, that we’re calling Copilot Workspace. Essentially, instead of starting right in the code, you start with a problem; you start with a GitHub issue describing what you’re trying to accomplish. Copilot can take that and go from prose [written prompts] of what you’re trying to do, into the code, then you can edit the prose and it’ll change the code. Then you can edit the code if you need to. Ultimately, it can test that, run that, build that and you can deploy it all starting from natural language.

“That’s an area we’re trying to push past and figure out how to write more code faster, more accurately. Sometimes the best code is no code at all and it’s just using human language.”

How do you determine ROI, and have you figured out an ROI on this technology? “You can measure a million things [internally]; that’s true for software development, too. In software development, there are practices where you can measure DORA metrics — this idea of how much code you’re pushing through your pipeline. You can measure space metrics. There are all kinds of methods here.

“Internally, when we were trying to work on these IT use cases, it was the same thing. How many tickets are deflected? How many tickets were closed? At the end of the day, we’re all measuring activity. Ultimately, what we’re trying to measure is time. How much time are you getting back?

“For me, when we’re doing all these measures, the ROI is baked in because we’re taking time back that previously employees were using to solve those tickets, do manual flows, write code — whatever that is. So, we take that time and invest it back into something that’s more strategic. So, the ROI has been pretty clear on this OctoBot idea; the 55% more productivity ultimately gets reinvested automatically because you’re able to move more quickly as a software developer, and then we’re trying out other tools that are giving back employees, potentially, 30 minutes to an hour every day. Then they get to do the work that’s more strategic for our customers and move more quickly.

“I think you can see that at GitHub how it feels like we’re able to do so much more than before, and it’s not because we’ve grown 50% in the last year; it’s because we’re able to invest that time back.

“So, for me I feel like every measure the industry tries to put on AI is us just trying to fit our scribes into a printing press modality and instead we all can agree if I had an hour every day where I could do something more that I love to do, that’s an easy measure. I can take that and invest it into what’s more strategic to me at GitHub.”

What tips do you have for companies considering using or scaling their use of genAI?  “The best thing companies can do to get started is find methods of folding AI into existing workflows — teams don’t need to reinvent the wheel, as it is easier to use AI functionalities that keeps you in the flow, not that requires net-new behavior. Meet employees where they are, give them the AI tools they need, and eventually they’ll evangelize it to others.

Some other tips stemming from my experience at GitHub include: 

• Make a plan you can repeat. This isn’t the era of ‘move fast and break thing,’ but it’s also not the time to develop a six-month, one-size-fits-all rollout strategy. But, we can learn from decades of writing software that iterating is more important than an enormous rewrite project that you hope is correct. It’s important to pilot often, early, in small groups. You need to make experimentation a core part of your culture. At GitHub, this looks like more than 10% of our company being involved in AI pilots.
• Let AI take your toil. The best AI tools are the ones that let our best human qualities shine. We don’t need AI to be creative while we toil to manipulate the AI to do what we want. Focus on what distracts employees from doing their jobs — like calendar management for example — and let AI take the toil so your employees can unleash their creativity in bigger, more profound ways.