Investigating the Efficacy of Large Language Models for Code Clone Detection (CCD)
The authors investigated the applicability of LLMs for CCD (Type-4 code clones).
RQs:
- What is the effect of different prompts to encourage Chat-GPT to identify Code Clones?
- What is the performance of ChatGPT for code clone detection compared to the baselines (CodeBERT, RoBERTa, GraphCodeBERT)?
ChatGPT (GPT-3.5-turbo) surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878.
The authors investigated the applicability of LLMs for CCD (Type-4 code clones).
RQs:
- What is the effect of different prompts to encourage Chat-GPT to identify Code Clones?
- What is the performance of ChatGPT for code clone detection compared to the baselines (CodeBERT, RoBERTa, GraphCodeBERT)?
ChatGPT (GPT-3.5-turbo) surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878.
Meta AI releases Code Llama 70B
The model is designed for general code synthesis and understanding.
Weights: https://huggingface.co/codellama/CodeLlama-70b-hf
The model is designed for general code synthesis and understanding.
Weights: https://huggingface.co/codellama/CodeLlama-70b-hf
X (formerly Twitter)
AI at Meta (@AIatMeta) on X
Today we’re releasing Code Llama 70B: a new, more performant version of our LLM for code generation — available under the same license as previous Code Llama models.
Download the models ➡️ https://t.co/fa7Su5XWDC
• CodeLlama-70B
• CodeLlama-70B-Python
•…
Download the models ➡️ https://t.co/fa7Su5XWDC
• CodeLlama-70B
• CodeLlama-70B-Python
•…
👍1
JetBrains' unremovable AI assistant meets irresistible outcry
Some JetBrains customers feel strongly about AI Assistant and really don't want the plugin to be present in their JetBrains applications at all, whether that's due to corporate policies that are incompatible with AI Assistant or other concerns. But because the plugin code has been "deeply integrated," removal has proven complicated.
More than dozen threads on JetBrains' YouTrack issue board have been posted seeking a way to delete, uninstall, or otherwise excise the AI Assistant plugin since it debuted.
Some JetBrains customers feel strongly about AI Assistant and really don't want the plugin to be present in their JetBrains applications at all, whether that's due to corporate policies that are incompatible with AI Assistant or other concerns. But because the plugin code has been "deeply integrated," removal has proven complicated.
More than dozen threads on JetBrains' YouTrack issue board have been posted seeking a way to delete, uninstall, or otherwise excise the AI Assistant plugin since it debuted.
😁2😱2👎1
GitBug-Java: A Reproducible Benchmark of Recent Java Bugs
The authors introduce GitBug-Java, a reproducible benchmark of recent Java bugs featuring 199 bug-fixes sourced from 55 relevant open-source repositories. To ensure the relevance of the bug-fixes to current development practices, the authors only collected bug-fixes from 2023. This may be useful in LLM evaluations.
GitBug-Java also provides offline reproduction environments for each collected bug-fix. To guarantee the validity and quality of the bug-fixes included in GitBug-Java, the authors manually curated the included bug-fixes.
site: https://www.nuno.saavedra.pt/gitbug-java/#!/
github: https://github.com/gitbugactions/gitbug-java
The authors introduce GitBug-Java, a reproducible benchmark of recent Java bugs featuring 199 bug-fixes sourced from 55 relevant open-source repositories. To ensure the relevance of the bug-fixes to current development practices, the authors only collected bug-fixes from 2023. This may be useful in LLM evaluations.
GitBug-Java also provides offline reproduction environments for each collected bug-fix. To guarantee the validity and quality of the bug-fixes included in GitBug-Java, the authors manually curated the included bug-fixes.
site: https://www.nuno.saavedra.pt/gitbug-java/#!/
github: https://github.com/gitbugactions/gitbug-java
StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
StepCoder is a novelty training framework for code generation via RL. It breaks down complicated exploration problems to reduce the difficulty of exploring environments with sparse rewards while providing fine-grained optimization. In addition, the authors constructed a high-quality dataset APPS+, specifically for code generation.
Dataset: https://github.com/Ablustrund/APPS_Plus
StepCoder is a novelty training framework for code generation via RL. It breaks down complicated exploration problems to reduce the difficulty of exploring environments with sparse rewards while providing fine-grained optimization. In addition, the authors constructed a high-quality dataset APPS+, specifically for code generation.
Dataset: https://github.com/Ablustrund/APPS_Plus
👍3
Collaborative Agents for Software Engineering
CodeAgent is a novel multi-agent-based system for code review. To demonstrate the effectiveness of CodeAgent, the authors performed experiments to assess its capabilities in various tasks including
- detection of inconsistencies between code changes and commit messages
- detection of vulnerability introduction by commits
- validation of adherence to code style.
Website: https://code-agent-new.vercel.app/index.html
CodeAgent is a novel multi-agent-based system for code review. To demonstrate the effectiveness of CodeAgent, the authors performed experiments to assess its capabilities in various tasks including
- detection of inconsistencies between code changes and commit messages
- detection of vulnerability introduction by commits
- validation of adherence to code style.
Website: https://code-agent-new.vercel.app/index.html
Mercury: An Efficiency Benchmark for LLM Code Synthesis
While LLMs demonstrate the remarkable capability to generate functionally correct code, there still exists a substantial gap in their efficiency output.
Mercury is a benchmark designated for assessing the code efficiency of LLM code synthesis tasks. It consists of 1,889 programming tasks covering diverse difficulty levels.
Each Mercury task includes a task denoscription, a test case generator, a prompt & entry point, and corresponding solutions.
github: https://github.com/Elfsong/Mercury
While LLMs demonstrate the remarkable capability to generate functionally correct code, there still exists a substantial gap in their efficiency output.
Mercury is a benchmark designated for assessing the code efficiency of LLM code synthesis tasks. It consists of 1,889 programming tasks covering diverse difficulty levels.
Each Mercury task includes a task denoscription, a test case generator, a prompt & entry point, and corresponding solutions.
github: https://github.com/Elfsong/Mercury
BAFLineDP: Code Bilinear Attention Fusion Framework for Line-Level Defect Prediction
The paper presents a line-level defect prediction method grounded in a code bilinear attention fusion framework (BAFN).
BAFN is designed to amalgamate global and local information by capturing the bilinear interaction attention weights between code lines and their respective line-level contextual information to construct defect code features.
github: https://github.com/insoft-lab/BAFLineDP
The paper presents a line-level defect prediction method grounded in a code bilinear attention fusion framework (BAFN).
BAFN is designed to amalgamate global and local information by capturing the bilinear interaction attention weights between code lines and their respective line-level contextual information to construct defect code features.
github: https://github.com/insoft-lab/BAFLineDP
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem
The paper presents a vision for the future of computing within the AIOS-Agent ecosystem, where the LLM functions as the core of AIOS (Artificial Intelligent Operating System).
The paper presents a vision for the future of computing within the AIOS-Agent ecosystem, where the LLM functions as the core of AIOS (Artificial Intelligent Operating System).
👍2
Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
Recently researchers proposed DoRA: Weight-Decomposed Low-Rank Adaptation. DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding
To understand how these methods work Sebastian Raschka in his article suggests implementing both LoRA and DoRA in PyTorch from scratch.
Recently researchers proposed DoRA: Weight-Decomposed Low-Rank Adaptation. DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding
To understand how these methods work Sebastian Raschka in his article suggests implementing both LoRA and DoRA in PyTorch from scratch.
👍3🔥2
Context-Aware Code Generation
In practice, you rarely need to write code taken out of context. As a rule, we write code that becomes part of the project. This code is closely related to the project ideologically, syntactically and stylistically. How to ensure this when generating code using LLM? If the project is small, then the entire project code can be passed as context. For a large project, this trick will not work, and such approaches as RAG (this one or this one) are required to fetch relevant information from existing code repositories and use it to create accurate code, documentation, or even fix code errors.
In practice, you rarely need to write code taken out of context. As a rule, we write code that becomes part of the project. This code is closely related to the project ideologically, syntactically and stylistically. How to ensure this when generating code using LLM? If the project is small, then the entire project code can be passed as context. For a large project, this trick will not work, and such approaches as RAG (this one or this one) are required to fetch relevant information from existing code repositories and use it to create accurate code, documentation, or even fix code errors.
🔥2
New Breakthrough Brings Matrix Multiplication Closer to Ideal
In 1969 a breakthrough result by Strassen showed that n x n matrices can be multiplied faster than the naive cubic time algorithm. Since then there has been an explosion of results obtaining lower and lower bounds on the exponent ω defined as the smallest constant such that for all ε > 0, n x n matrices can be multiplied using O(n^{ω+ε}) arithmetic operations.
The new bound on ω is ω ≤ 2.371552 .
paper: https://epubs.siam.org/doi/10.1137/1.9781611977912.134
In 1969 a breakthrough result by Strassen showed that n x n matrices can be multiplied faster than the naive cubic time algorithm. Since then there has been an explosion of results obtaining lower and lower bounds on the exponent ω defined as the smallest constant such that for all ε > 0, n x n matrices can be multiplied using O(n^{ω+ε}) arithmetic operations.
The new bound on ω is ω ≤ 2.371552 .
paper: https://epubs.siam.org/doi/10.1137/1.9781611977912.134
🔥4