Examining the Feasibility and Influencing Factors of Large Language Models in Source Code Security Analysis
Saved in:
| Title: | Examining the Feasibility and Influencing Factors of Large Language Models in Source Code Security Analysis |
|---|---|
| Authors: | Lin, Jie |
| Committee Members: | Mohaisen, David |
| Summary: | Large Language Models (LLMs) have recently emerged as promising tools for analyzing source code; however, their capabilities for automated vulnerability detection and localization remain underexplored, particularly given the widespread reliance on closed-source models which introduce significant privacy, security, and transparency risks. This dissertation rigorously examines the feasibility and key influencing factors determining the efficacy of open-source LLMs in source code security analysis. Initially, we investigate the impact of tokenized input length on vulnerability detection accuracy and explicitness across Java vulnerability datasets using ten distinct LLM architectures, identifying robust performance in specific models while others demonstrate considerable accuracy deterioration. Subsequently, we comprehensively evaluate the true vulnerability detection effectiveness of 38 open-source LLM configurations using both vulnerable and non-vulnerable datasets spanning Java and C/C++, applying realistic zero-shot and few-shot prompting scenarios and meticulously assessing model performance by specific vulnerability types. Furthermore, extending beyond mere detection capabilities, we conduct an extensive empirical study involving 66 LLM configurations for Java and 63 for C/C++ to examine hallucination behaviors during vulnerability localization. Our findings highlight that structured, assumption-based prompts substantially mitigate hallucinations, particularly among mid-sized models; nevertheless, precise vulnerability localization remains notably challenging, especially within C/C++ contexts. Finally, by evaluating precision, recall, and F1 metrics for various open-source LLMs at the function-level localization task, we systematically characterize their true effectiveness and practical applicability. Collectively, this dissertation establishes that through strategic prompt engineering, meticulous model selection, and rigorous empirical validation, open-source LLMs can reliably, transparently, and securely enhance vulnerability detection and localization processes, thereby substantially contributing to secure software development practices. |
| URL: | https://stars.library.ucf.edu/etd2024/340 |
| Database: | OpenDissertations |
| Abstract: | Large Language Models (LLMs) have recently emerged as promising tools for analyzing source code; however, their capabilities for automated vulnerability detection and localization remain underexplored, particularly given the widespread reliance on closed-source models which introduce significant privacy, security, and transparency risks. This dissertation rigorously examines the feasibility and key influencing factors determining the efficacy of open-source LLMs in source code security analysis. Initially, we investigate the impact of tokenized input length on vulnerability detection accuracy and explicitness across Java vulnerability datasets using ten distinct LLM architectures, identifying robust performance in specific models while others demonstrate considerable accuracy deterioration. Subsequently, we comprehensively evaluate the true vulnerability detection effectiveness of 38 open-source LLM configurations using both vulnerable and non-vulnerable datasets spanning Java and C/C++, applying realistic zero-shot and few-shot prompting scenarios and meticulously assessing model performance by specific vulnerability types. Furthermore, extending beyond mere detection capabilities, we conduct an extensive empirical study involving 66 LLM configurations for Java and 63 for C/C++ to examine hallucination behaviors during vulnerability localization. Our findings highlight that structured, assumption-based prompts substantially mitigate hallucinations, particularly among mid-sized models; nevertheless, precise vulnerability localization remains notably challenging, especially within C/C++ contexts. Finally, by evaluating precision, recall, and F1 metrics for various open-source LLMs at the function-level localization task, we systematically characterize their true effectiveness and practical applicability. Collectively, this dissertation establishes that through strategic prompt engineering, meticulous model selection, and rigorous empirical validation, open-source LLMs can reliably, transparently, and securely enhance vulnerability detection and localization processes, thereby substantially contributing to secure software development practices. |
|---|
