Reference material

The issue

One the open issues in arkanjo repository was the following:

[Low Priority] The tool supports two methods for detection: an NLP similarity metric using Gensim and a simple arithmetic formula based on the results from the diff command. If someone wants, they can try adding more state-of-the-art techniques to detect code duplication, or at least use an LLM for similarity detection instead of simple NLP techniques. Note that state-of-the-art techniques tend to be considerably slower than other methods, so they will only work on small codebases. Code pointer: pre/preprocessor.cpp

The detection method originally implemented in ArKanjo relies on a classic NLP approach: each function is reduced to a bag of tokens and weighted with TF-IDF (Term Frequency–Inverse Document Frequency), after which the similarity between two functions is computed as the cosine of the angle between their vectors.

The project’s creator deliberately chose TF-IDF because the primary goal was to evaluate an extremely large codebase—the Linux kernel—and this method is computationally cheap and highly scalable, making it feasible to preprocess and compare millions of functions in a reasonable amount of time. However, this performance comes at the cost of accuracy and depth. Because TF-IDF treats code as an unordered collection of words, it is purely lexical: it ignores the syntactic structure, control flow, and—most importantly—the semantics of the code. As a result, it tends to flag functions that merely share common tokens while missing genuine semantic clones (functionally equivalent code written with different identifiers or constructs). These limitations are what motivated the issue.

The New method

To add a new option to detect duplicated code we created a python script using jina embedding model. The script was added in third-party section, with its own requirementes and doc.

A new function-level duplication detector that measures semantic similarity between function bodies was added. Instead of comparing tokens, lines, or term frequencies, it embeds each function with a Hugging Face model (jinaai/jina-embeddings-v2-base-code) and computes the pairwise cosine similarity between the resulting vectors, scaled to a 0–100 range. Because the comparison happens in embedding space rather than over surface text, the method can flag functions that do the same thing even when their names, formatting, or syntax differ.

The detector is implemented in two layers. On the C++ side, the new LLMMethod class hooks into the existing preprocessing pipeline: for every function it materializes the function body into its own source file under the cache directory, then invokes a Python script (llm_detection.py) as a subprocess and reads the resulting duplication pairs back through a pipe, filtering them by the similarity threshold and saving them in ArKanjo’s standard output format.

The Python script does the actual embedding work using the sentence-transformers library: it loads the model, encodes every function body into a normalized vector, computes all pairwise cosine similarities at once as a single matrix product (the dot product of normalized vectors is the cosine), scales each score to 0–100, and emits every pair that meets the minimum-similarity threshold.

After the core method was in place, additional changes were made to expose its behavior through command-line arguments, since embedding a large codebase is far more memory-intensive than the previous frequency-based approach. The new flags let the user tune this trade-off:

• –llm-max-seq-length — caps the number of tokens considered per function body (default 1024; the model accepts up to 8192).Attention memory grows roughly as O(seq_len²), so this is the main lever for bounding peak memory; longer bodies are truncated.
• –llm-batch-size — how many function bodies are embedded together in a single batch (default 4). Lowering it reduces memory usage on machines with less RAM at the cost of throughput.
• –llm-model — overrides the Hugging Face sentence-transformers model used for the embeddings, allowing other code-aware models to be plugged in without code changes.

Executing the new method

The embedding-based detector relies on Python libraries (sentence-transformers, transformers, numpy, and einops) that are not needed by the other methods. To keep these dependencies isolated from the system Python, we use a virtual environment.

Create the virtual environment:

python3 -m venv .venv

Activate it so the method runs against the isolated interpreter:

source .venv/bin/activate

Install the Python dependencies required by the detector:

pip install -r third-party/llm-detection/requirements.txt

On the first run the chosen model (jinaai/jina-embeddings-v2-base-code) is downloaded automatically from Hugging Face and cached locally, so this step requires an internet connection.

With the environment ready, run the preprocessor:

arkanjo-preprocessor build

The preprocessor will interactively ask for the path to the codebase, the similarity threshold, and the duplication-finder technique to use. Choose option 4 — Embedding-based similarity using a code language model to select the new method.

Optionally, you can tune the detector’s memory/performance trade-off by passing any of the LLM-specific flags:

arkanjo-preprocessor build --llm-max-seq-length 1024 --llm-batch-size 4 --llm-model jinaai/jina-embeddings-v2-base-code

Testing

Regression testing

Before evaluating the new method, we first ran a regression test to confirm that the three existing methods still worked correctly after our changes, so that any difference in behavior could be attributed to the new code and not to a regression.

Choosing an evaluation dataset

Our next step was to evaluate the embedding-based method and compare its results against the existing ones. We initially intended to reuse the benchmark from the tool’s original evaluation, but we ran into two problems: it was difficult to set up and reuse, and we found it incomplete, since it only reported recall and therefore could not show whether a method also produced many false positives. For a fair comparison we needed a dataset that supported a fuller set of metrics, so we looked for an alternative.

We chose the POJ-104 competitive programming dataset:

• https://arxiv.org/abs/1409.5718
• https://huggingface.co/datasets/google/code_x_glue_cc_clone_detection_poj104/viewer

In this dataset each folder groups different solutions to the same competitive programming problem, which gives a natural ground truth for clone detection: programs in the same folder are clones of one another, and programs in different folders are not.

Adapting the tool and the dataset to work together

Using this dataset required several changes, since it did not match what ArKanjo expected:

• Unsupported file extension. The dataset’s files used the .txt extension, which ArKanjo does not recognize.
• Unsupported language. The programs were written in C++ (not supported by ArKanjo at the time) and C. We added support for C++ and created a new version of the dataset with every file renamed to the .cpp extension, which also works for the C files.
• Comparison granularity. By default ArKanjo splits each file into individual functions, but in this dataset a whole program is the unit being compared; comparing isolated functions would not be meaningful. To handle this we added an option to run the tool at file granularity, keeping each program whole.

Handling memory limitations

Our first attempt was to run the method over the entire dataset, but this exhausted memory and crashed the application. Investigating the failure, we found that the memory pressure came largely from embedding multiple function bodies in parallel. This is what motivated the customizable parameters (–llm-batch-size and –llm-max-seq-length), which let the user bound peak memory usage. To keep the evaluation tractable we also worked with a sample of the dataset: 30 folders with 10 programs each.

Fixing duplicated results

During testing we also noticed that the embedding-based method was reporting the same pair of functions more than once in its results. We tracked this down to the comparison loop and fixed it so that each pair is emitted only once.

Trade-offs observed

In our experiments the embedding-based method proved slower than the original NLP method, but, unlike the NLP method — which crashed on the full dataset — our final embedding-based solution ran without memory problems once the tuning parameters were in place.

Metrics

To evaluate and compare the methods on this dataset we computed recall, precision, F1, and accuracy — a more complete picture than the recall-only original benchmark.

The evaluation was run with file granularity, for example:

arkanjo-preprocessor build --name nlp-programs-dataset --granularity file

The new method had betters results when compared to the other 3 methods.

The chart below summarizes the comparison. For each of the four detection methods we plot recall, precision, F1, and accuracy on the POJ-104 sample (30 folders with 10 programs each), where same-folder programs are clones and different-folder programs are not. The embedding-based detector (jina-embeddings-v2-base-code) comes out ahead of the three existing methods: by comparing functions in embedding space rather than over surface tokens, it catches semantic clones that the lexical TF-IDF approach misses while keeping false positives low, which is reflected in its higher precision and F1. This is the central result of the work — semantic similarity detection outperforms the classic NLP and arithmetic methods on this dataset, at the cost of being slower.