Toward a Causal Understanding of Neural Language Models for Code Generation.
The combination of large amounts of freely available code-related data, which can be mined from open source repositories, and ever-more sophisticated \textit{Neural Language Model} (NLM) architectures have fueled the development of Software Engineering (SE) tools with increasing effectiveness. NLMs for code have (seemingly) illustrated promising performance across a range of different SE tasks (Watson et al., 2020; White et al., 2020; Ciniselli et al., 2021; Mastropaolo et al., 2021). In particular, \textit{code generation} has been an important area of SE research for decades, enabling tools for downstream tasks such as code completion~\cite{MSR-Completion}, program repair (Chen et al., 2019), and test case generation (Watson et al., 2020). In addition, industry interest in leveraging NLMs for code generation has also grown as evidenced by tools just as Microsoft’s IntelliCode \cite{intellicode}, Tabnine \cite{tabnine}, OpenAI’s Codex \cite{openai_codex}, and GitHub’s Copilot \cite{github_copilot}. Given the prior popularity of code completion engines within IDEs~\cite{murphy2006ide}, and the pending introduction of, and investment in commercial tools, NLMs for code generation will almost certainly be used to help build production software systems in the near future, if they are not already being used.