publications

2024

1. ArXiv

   Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations

   David N. Palacio , Daniel Rodriguez-Cardenas , Alejandro Velasco , and 3 more authors

   2024

   HTML
2. TSE

   Toward a Theory of Causation for Interpreting Neural Code Models

   David N. Palacio , Alejandro Velasco , Nathan Cooper , and 3 more authors

   IEEE Transactions on Software Engineering, 2024

   Abs arXiv HTML Code

   Neural Language Models of Code, or Neural Code Models (NCMs), are rapidly progressing from research prototypes to commercial developer tools. As such, understanding the capabilities and limitations of such models is becoming critical. However, the abilities of these models are typically measured using automated metrics that often only reveal a portion of their real-world performance. While, in general, the performance of NCMs appears promising, currently much is unknown about how such models arrive at decisions. To this end, this paper introduces docode, a post hoc interpretability method specific to NCMs that is capable of explaining model predictions. docode is based upon causal inference to enable programming language-oriented explanations. While the theoretical underpinnings of docode are extensible to exploring different model properties, we provide a concrete instantiation that aims to mitigate the impact of spurious correlations by grounding explanations of model behavior in properties of programming languages. To demonstrate the practical benefit of docode, we illustrate the insights that our framework can provide by performing a case study on two popular deep learning architectures and ten NCMs. The results of this case study illustrate that our studied NCMs are sensitive to changes in code syntax. All our NCMs, except for the BERT-like model, statistically learn to predict tokens related to blocks of code (\eg brackets, parenthesis, semicolon) with less confounding bias as compared to other programming language constructs. These insights demonstrate the potential of docode as a useful method to detect and facilitate the elimination of confounding bias in NCMs.

3. ICSE

   Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?

   Alejandro Velasco , David N. Palacio , Daniel Rodríguez-Cárdenas , and 1 more author

   ICSE, 2024

   arXiv HTML Code
4. PATENT

   Patent: Debugging Tool for Code Generation Neural Language Models

   2024

   HTML Website

2023

1. ICSME

   Benchmarking Causal Study to Interpret Large Language Models for Source Code

   Daniel Rodriguez-Cardenas , David N. Palacio , Dipin Khati , and 2 more authors

   2023

   arXiv HTML Code
2. TSE

   Using Transfer Learning for Code-Related Tasks

   Antonio Mastropaolo , Nathan Cooper , David Nader Palacio , and 4 more authors

   2023

   Abs HTML

   Deep learning (DL) techniques have been used to support several code-related tasks such as code summarization and bug-fixing. In particular, pre-trained transformer models are on the rise, also thanks to the excellent results they achieved in Natural Language Processing (NLP) tasks. The basic idea behind these models is to first pre-train them on a generic dataset using a self-supervised task (e.g., filling masked words in sentences). Then, these models are fine-tuned to support specific tasks of interest (e.g., language translation). A single model can be fine-tuned to support multiple tasks, possibly exploiting the benefits of transfer learning. This means that knowledge acquired to solve a specific task (e.g., language translation) can be useful to boost performance on another task (e.g., sentiment classification). While the benefits of transfer learning have been widely studied in NLP, limited empirical evidence is available when it comes to code-related tasks. In this paper, we assess the performance of the Text-To-Text Transfer Transformer (T5) model in supporting four different code-related tasks: (i) automatic bug-fixing, (ii) injection of code mutants, (iii) generation of assert statements, and (iv) code summarization. We pay particular attention in studying the role played by pre-training and multi-task fine-tuning on the model’s performance. We show that (i) the T5 can achieve better performance as compared to state-of-the-art baselines; and (ii) while pre-training helps the model, not all tasks benefit from a multi-task fine-tuning.

3. ArXiv

   Evaluating and Explaining Large Language Models for Code Using Syntactic Structures

   David N. Palacio , Alejandro Velasco , Daniel Rodriguez-Cardenas , and 2 more authors

   2023

   Abs arXiv HTML

   Large Language Models (LLMs) for code are a family of high-parameter, transformer-based neural networks pre-trained on massive datasets of both natural and programming languages. These models are rapidly being employed in commercial AI-based developer tools, such as GitHub CoPilot. However, measuring and explaining their effectiveness on programming tasks is a challenging proposition, given their size and complexity. The methods for evaluating and explaining LLMs for code are inextricably linked. That is, in order to explain a model’s predictions, they must be reliably mapped to fine-grained, understandable concepts. Once this mapping is achieved, new methods for detailed model evaluations are possible. However, most current explainability techniques and evaluation benchmarks focus on model robustness or individual task performance, as opposed to interpreting model predictions. To this end, this paper introduces ASTxplainer, an explainability method specific to LLMs for code that enables both new methods for LLM evaluation and visualizations of LLM predictions that aid end-users in understanding model predictions. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes, by extracting and aggregating normalized model logits within AST structures. To demonstrate the practical benefit of ASTxplainer, we illustrate the insights that our framework can provide by performing an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects. Additionally, we perform a user study examining the usefulness of an ASTxplainer-derived visualization of model predictions aimed at enabling model users to explain predictions. The results of these studies illustrate the potential for ASTxplainer to provide insights into LLM effectiveness, and aid end-users in understanding predictions.

2022

1. TOSEM

   A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research

   Cody Watson , Nathan Cooper , David Nader Palacio , and 2 more authors

   ACM Trans. Softw. Eng. Methodol., Mar 2022

   Abs arXiv HTML

   An increasingly popular set of techniques adopted by software engineering (SE) researchers to automate development tasks are those rooted in the concept of Deep Learning (DL). The popularity of such techniques largely stems from their automated feature engineering capabilities, which aid in modeling software artifacts. However, due to the rapid pace at which DL techniques have been adopted, it is difficult to distill the current successes, failures, and opportunities of the current research landscape. In an effort to bring clarity to this cross-cutting area of work, from its modern inception to the present, this article presents a systematic literature review of research at the intersection of SE & DL. The review canvasses work appearing in the most prominent SE and DL conferences and journals and spans 128 papers across 23 unique SE tasks. We center our analysis around the components of learning, a set of principles that governs the application of machine learning techniques (ML) to a given problem domain, discussing several aspects of the surveyed work at a granular level. The end result of our analysis is a research roadmap that both delineates the foundations of DL techniques applied to SE research and highlights likely areas of fertile exploration for the future.

2020

1. ICSE

   Improving the effectiveness of traceability link recovery using hierarchical bayesian networks

   Kevin Moran , David N. Palacio , Carlos Bernal-Cárdenas , and 4 more authors

   In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , Mar 2020

   Abs HTML

   Traceability is a fundamental component of the modern software development process that helps to ensure properly functioning, secure programs. Due to the high cost of manually establishing trace links, researchers have developed automated approaches that draw relationships between pairs of textual software artifacts using similarity measures. However, the effectiveness of such techniques are often limited as they only utilize a single measure of artifact similarity and cannot simultaneously model (implicit and explicit) relationships across groups of diverse development artifacts.

2. Subreviewers

   In 2020 IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems (MOBILESoft) , Mar 2020

2019

1. ICSME

   Learning to Identify Security-Related Issues Using Convolutional Neural Networks

   David N. Palacio , Daniel McCrystal , Kevin Moran , and 3 more authors

   Mar 2019

   Abs HTML Code

   Software security is becoming a high priority for both large companies and start-ups alike due to the increasing potential for harm that vulnerabilities and breaches carry with them. However, attaining robust security assurance while delivering features requires a precarious balancing act in the context of agile development practices. One path forward to help aid development teams in securing their software products is through the design and development of security-focused automation. Ergo, we present a novel approach, called SecureReqNet, for automatically identifying whether issues in software issue tracking systems describe security-related content. Our approach consists of a two-phase neural net architecture that operates purely on the natural language descriptions of issues. The first phase of our approach learns high dimensional word embeddings from hundreds of thousands of vulnerability descriptions listed in the CVE database and issue descriptions extracted from open source projects. The second phase then utilizes the semantic ontology represented by these embeddings to train a convolutional neural network capable of predicting whether a given issue is securityrelated. We evaluated SecureReqNet by applying it to identify security-related issues from a dataset of thousands of issues mined from popular projects on GitLab and GitHub. In addition, we also applied our approach to identify security-related requirements from a commercial software project developed by a major telecommunication company. Our preliminary results are encouraging, with SecureReqNet achieving an accuracy of 96% on open source issues and 71.6% on industrial requirements.

2. Additional Reviewers

   In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) , Mar 2019

2018

1. GECCO

   Assessing Single-Objective Performance Convergence and Time Complexity for Refactoring Detection

   David Nader-Palacio , Daniel Rodrı́guez-Cárdenas , and Jonatan Gomez

   In Proceedings of the Genetic and Evolutionary Computation Conference Companion , Kyoto, Japan, Mar 2018

   Abs HTML

   The automatic detection of refactoring recommendations has been tackled in prior optimization studies involving bad code smells, semantic coherence and importance of classes; however, such studies informally addressed formalisms to standardize and replicate refactoring models. We propose to assess the refactoring detection by means of performance convergence and time complexity. Since the reported approaches are difficult to reproduce, we employ an Artificial Refactoring Generation (ARGen) as a formal and naive computational solution for the Refactoring Detection Problem. ARGen is able to detect massive refactoring’s sets in feasible areas of the search space. We used a refactoring formalization to adapt search techniques (Hill Climbing, Simulated Annealing and Hybrid Adaptive Evolutionary Algorithm) that assess the performance and complexity on three open software systems. Combinatorial techniques are limited in solving the Refactoring Detection Problem due to the relevance of developers’ criteria (human factor) when designing reconstructions. Without performance convergence and time complexity analysis, a software empirical analysis that utilizes search techniques is incomplete.

2017

1. A computational solution for the software refactoring problem: from a formalism toward an optimization approach

   Nader Palacio , and David Alberto

   Dec 2017

   HTML