We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 – 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!
GitHub announced last week that it will be releasing Copilot, its “AI pair programmer” tool, to the public. Copilot uses AI to provide a range of support functions, including autocompleting instructions, generating entire functions, and transforming docstrings and descriptions into functional source code.
Copilot launched as a technical preview in 2021. Now all developers can apply for Copilot, which installs as an extension in integrated development environments (IDE) such as Visual Studio, VS Code, Neovim and JetBrains IDEs.
At the time of Copilot’s release, there was a lot of excitement around its stunning coding capabilities. But there were also concerns about how far its abilities can be trusted and whether it has a real impact on the productivity of developers. After a year and billions of lines of code, Copilot is finally ready to be in the hands of every developer.
Here’s what we know about Copilot’s effect on real programming tasks, told by its creators and developers who have used it in their day-to-day work.
Event
Transform 2022
Join us at the leading event on applied AI for enterprise business and technology decision makers in-person July 19 and virtually from July 20-28.
How much code is written with Copilot?
Behind Copilot is the transformer architecture, the kind of deep learning model used in large language models such as GPT-3 and LaMDA. Transformers are especially good at processing sequential data such as text, software code and protein sequences. Given a prompt, a transformer model can predict the next elements of the sequence, whether it is words or computer instructions. Copilot is built on OpenAI’s Codex, a transformer that has been trained on tens of millions of code repositories. Once installed on your IDE, Copilot provides suggestions based on the existing code in your file as well as cues, such as the names of your functions and classes and the comments in your code.
It is worth noting that Copilot does not think and code like a programmer. But since it has been exposed to huge amounts of source code, it can provide very good code suggestions, especially on standard and repetitive tasks that occur frequently in writing software.
According to GitHub, more than 1.2 million developers used Copilot’s technical preview in the past 12 months. In files where Copilot is enabled, it accounts for nearly 40% of code in popular programming languages like Python.
Ryan J. Salva, VP of product at GitHub, told VentureBeat that while it is hard to say how much of the coding done with Copilot is real software development as opposed to exploring the tool, the 40% ratio seemed to hold as developers used Copilot over several days.
Github Copilot reduces distractions
“Developers often talk about the value of staying ‘in the flow’ and the positive impact [Copilot] has on their productivity, creativity and overall happiness,” Salva said.
According to Salva, in a survey distributed to 17,000 developers during Copilot’s technical preview, over 75% of developers self-reported that when using Copilot they “spend less mental effort on repetitive programming tasks,” “focus on more satisfying work” and “stay in the flow.”
“By minimizing distractions and creating focus time, we not only get work done, we create better and less stressful days,” Salva said. “Anecdotally, we’ve heard stories of developers using Copilot to learn new coding languages, quickly generate boilerplate code for common tasks, write regular expressions or simply recall the syntax for an API without needing to consult documentation.”
Developers VentureBeat spoke to confirm some of these points. Abhishek Thakur, a machine learning engineer at Hugging Face, has been using Copilot since June 2021. He has used it in developing AutoTrain, a no-code tool for training state-of-the-art machine learning models. He also uses it for machine learning competitions on Kaggle, making tutorials and participating in hackathons.
“When I am coding, I want the least distractions. In that way, Copilot has been a huge help. It has reduced the time I might spend looking for solutions on the web and instead have them at my fingertips in my favorite IDE,” Thakur said.
Many developers search for solutions to small problems on search engines and StackOverflow, a web forum where developers share code snippets for specific tasks.
“After using Copilot, I rarely visit these websites and can rather focus on coding,” Thakur said.
“If I am in doubt, I try to write comments and let Copilot help me finish the code chunks,” Thakur said. “It might not always be perfect, but it gives a good idea of how the code can be written. The rest is up to the developer to modify and reuse. Same is true for StackOverflow: It doesn’t always have the answer but it does have a lot of good answers which might suit your use case, and you can modify and reuse.”
Tackling repetitive tasks with Copilot
Louis Castricato, a research intern at Hugging Face and previously at EleutherAI, has used Copilot for scientific computing, where functions are often cumbersome and hard to use. One of these functions is PyTorch’s einsum, which requires unwieldy parameters about the dimensions of tensors that you want to compute.
“Einsum is very nonintuitive to people who have never used it before, and it requires you to pay close attention to the shape that your tensors are taking at every instruction within a call to your model,” Castricato said. “Copilot is particularly strong at inferring the shape of a tensor and automatically writing einsum operations, as well as writing comments explaining the choices it made in writing the einsum expression.”
Snir Shechter, R&D team lead at Aporia, has also been using Copilot for nearly a year. “When developing our main product at Aporia, Copilot helps me with writing the easy code,” he said. “Given good naming conventions, Copilot is able to complete the whole function/next block of code. After that, I just need to review to see that all is good (and possibly add more specific logic). It’s really good with completing generic/repetitive code and it figures it out based on the context.”
Copilot’s performance particularly stands out for lengthy and repetitive tasks. An example is launching an HTTP server, which usually requires several lines of code and adjustments, depending on the language it is being written in. In one study, GitHub required half of the participants to write the HTTP server code manually and the other half to complete the task using Copilot.
“Preliminary data suggests that developers are not only more likely to complete their task when using Copilot, but they also do it in roughly half the time,” Salva said.
Pushing developers to better document code
Copilot works better when programmers provide it with more detailed descriptions. Interestingly, using it has pushed developers to better document their code.
“In the first few months of the technical preview, we’ve seen Copilot change people’s behavior when writing code – namely by writing better, more verbose comments,” Salva said. “This is not only so that Copilot suggestions improve, but it makes it easier to read for others.”
Copilot has also become a good tool for documenting software code, a task that is often overlooked, especially when programmers are chasing deadlines.
Castricato uses Copilot to document his code, autocompleting docstrings and type suggestions in Python. This improves the readability of the code and makes it easier for himself and other developers to manage the code later.
“Copilot has increased the amount of documentation I write for my code by at least 2x or 3x,” he said.
The limits of Copilot
“[Copilot] is often quite poor at implementing entire algorithms,” Castricato said. “For instance, when I first got Copilot, I wanted to see if it could implement basic forms of dynamic programming without significant guidance. It failed miserably, and I very quickly realized that in order to use Copilot to its fullest ability, you need to explain (through comments) in detail the steps that Copilot needs to take to implement a certain algorithm.”
Beyond basic tasks, Copilot will need ample comments to function properly. And in some cases, it will need a fully structured code file to provide useful suggestions.
“In this regard, Copilot is very far away from replacing even the most rudimentary of software engineers,” Castricato said.
Salva acknowledged that Copilot is still a work in progress and a new developer experience. The product team continues to learn lessons from how developers use it and are adjusting the AI model that powers it.
“Copilot tries to understand your intent and to generate the best code it can, but the code it suggests may not always work or even make sense,” Salva said. “While we are working hard and seeing progress in Copilot generating better code, suggestions should be carefully tested, reviewed and vetted, like any other code. We are collecting telemetry data to make the model better, which we prompt users with in the UI.”
Is Copilot worth the price?
For the moment, Copilot will be offered at $10 per month, or $100 per year, with a 60-day free trial, which seems to be a bargain for software developers. In addition, the tool will be free for students and maintainers of popular open-source projects.
“I think it’s fully worth the price tag,” Thakur said. “As a machine learning engineer, I know that a lot goes into building products like these, especially Copilot, which provides suggestions with sub-milliseconds latency. To build an infrastructure that serves these kinds of models for free is not feasible in the real world for a longer period of time.”
Thakur also noted that as the costs of AI infrastructure continue to reduce, the price of Copilot may get lower in the future.
“But at this point, in my opinion, it’s totally worth the price, especially for someone like me who has been using Copilot almost every day since the preview,” he said.
Castricato said that Copilot has saved him multiple hours per week, and sometimes even a day or two of troubleshooting per week.
“Copilot allows me to rapidly test many experiments — often without having to spend extensive time debugging. A set of experiments that conventionally would take me days to implement instead takes me one day,” he said. “As a professional tool, it is well worth its price. It certainly makes me more than $10 a month. I can easily see any large company justifying a Copilot license for all of their technical staff. It’s almost a trivial expense.”
Salva believes that this is just the beginning of AI-augmented programming and sees Copilot as the next step in a long line of developer tools.
“As we saw with the compiler, higher-level programming languages and open source itself, tooling advancements have amplified the impact developers have in our world,” he said. “At the same time, those same tools are no substitute for a developer’s experience, skill and creativity.”
With better tools, he added, industry demand for developers has steadily increased. “We’re optimistic that GitHub Copilot will have similar effects, complementing the labor of developers and empowering them to write code more easily with greater focus and creativity,” he said.
VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.
Source: Read Full Article