Article · Cybersecurity

Using an Open-Source AI Model to Detect Camouflaged Attacks: A Threshold-Tuning Guide

Caleb Ngeno

By · November 17, 2026

Plain server room with rows of network cabinets

Abstract. An open-source model can help detect camouflaged attacks, but the model is only half of the system. The other half is the threshold that decides when a score becomes an alert, a quarantine, or no action. This article shows how to tune that threshold with a labeled evaluation set, hard negatives, calibrated scores, and a cost model. It uses BGE-M3, an MIT-licensed embedding model, with a calibrated classifier and a small novelty score. The result is a two-level policy: a high threshold for automatic action and a low threshold for human review. The article gives code, explains how to test against attack families held out from training, and shows how to monitor drift. The main point is practical. A threshold is not universal. It is a decision rule that belongs to one environment, one risk appetite, and one review capacity.

Keywords: open-source AI model, attack detection, threshold tuning, camouflaged attacks, BGE-M3, calibrated classifier, anomaly detection, prompt injection, threat detection, security analytics

Camouflaged attacks are hard because they copy the shape of normal work. A malicious instruction hides inside a support ticket. A credential dump looks like a backup job. An indirect prompt injection sits inside a document that an agent reads. A rare but valid admin action appears beside a familiar service account.

A model can read those patterns. A threshold decides what the team does about them. Set it too low and analysts drown in alerts. Set it too high and the detector misses the attack it was built to catch.

The goal is not a magic number. The goal is a threshold you can defend with test results.

What the detector should produce

Start with the decision you want the system to make. A useful detector returns a score between 0 and 1, where a higher score means higher risk. It also returns the reason for the score, such as a matched attack pattern, a high novelty value, or both.

Use three outcomes:

  • Below the low threshold: allow the action and keep the log.
  • Between the low and high thresholds: send it to a human queue.
  • Above the high threshold: quarantine, block, or step up authentication.

This design gives the team room to make mistakes. Automatic blocking needs strong evidence. Human review can handle softer signals.

Choose a model you can inspect

BGE-M3 is a good starting point for text-heavy detection. It is open source, MIT licensed, multilingual, and handles long context. You can run it locally, so event text does not need to leave your environment. The project publishes the model card and license on Hugging Face.

BGE-M3 does not produce a risk score by itself. It turns text into embeddings. Add a small calibrated classifier on top, then add a novelty score for events that look far from benign traffic. This keeps the system explainable enough for security work.

Qwen2.5-7B-Instruct is Apache 2.0 licensed and works well as a second-stage analyst assistant. It can summarize why an event scored high. Keep it out of the final decision path until you have evaluated it. A language model that writes a good explanation can still be a poor judge.

Build the evaluation set first

Collect event summaries from logs, alerts, tickets, agent tool calls, and identity records. Normalize each event into one text field. Keep a family label so you can hold out whole attack types during testing.

A simple record looks like this:

json
{
  "event_id": "evt-1042",
  "actor": "svc_backup",
  "action": "read",
  "resource": "prod/db/orders",
  "text": "backup service read orders table from new subnet after token refresh",
  "family": "benign",
  "label": 0
}

Then add hard negatives. These are benign events that look like attacks:

  • A security team running an authorized scan.
  • A backup job reading many tables at night.
  • A developer using a new VPN region.
  • An admin changing a mailbox rule during an incident.
  • A document that mentions credentials in a training guide.

Add camouflaged positives too:

  • Prompt injection inside a retrieved document.
  • A script renamed to match a maintenance job.
  • A suspicious command wrapped in routine deployment text.
  • An account takeover that follows normal working hours.
  • A rare action from a service account with a familiar name.

If the evaluation set only contains obvious cases, the threshold will look excellent in a notebook and fail in production.

Turn text into calibrated scores

Install the dependencies:

bash
pip install sentence-transformers scikit-learn pandas numpy

The following script embeds text, fits a calibrated classifier, and saves the model. It uses a stratified split for the first pass. Later, use a family-held-out split for the real test.

python
import joblib
import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

MODEL_NAME = "BAAI/bge-m3"

df = pd.read_json("events.jsonl", lines=True)
train_text, test_text, train_y, test_y = train_test_split(
    df["text"].astype(str),
    df["label"].astype(int),
    test_size=0.25,
    stratify=df["label"],
    random_state=42,
)

encoder = SentenceTransformer(MODEL_NAME)
train_x = encoder.encode(train_text.tolist(), normalize_embeddings=True)
test_x = encoder.encode(test_text.tolist(), normalize_embeddings=True)

base = LogisticRegression(
    C=1.0,
    class_weight="balanced",
    max_iter=2000,
)
model = CalibratedClassifierCV(base, method="sigmoid", cv=5)
model.fit(train_x, train_y)

probabilities = model.predict_proba(test_x)[:, 1]
joblib.dump(model, "attack_classifier.joblib")
pd.DataFrame({
    "text": test_text.values,
    "label": test_y.values,
    "score": probabilities,
}).to_csv("scores.csv", index=False)

Calibration matters because a threshold only means something when the score has a consistent meaning. A raw model output of 0.80 may behave like 0.40 on another data set. A calibrated score gives the team a stable operating language.

Add a novelty signal

Camouflaged attacks often look unlike any approved example. A classifier trained on past attacks can miss them. A novelty score catches part of that gap.

Compute the average embedding of benign training events. Then measure cosine distance from each new event to that center. Calibrate the distance against benign validation data and convert it into a 0 to 1 score.

python
import numpy as np
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("BAAI/bge-m3")
benign_texts = [
    "backup service read orders table after token refresh",
    "developer opened staging logs from approved vpn",
    "admin changed mailbox rule during incident review",
]
benign_x = encoder.encode(benign_texts, normalize_embeddings=True)
center = benign_x.mean(axis=0)
center = center / np.linalg.norm(center)

new_texts = [
    "maintenance job exported private keys to a new bucket",
    "support agent opened attachment with hidden instruction",
]
new_x = encoder.encode(new_texts, normalize_embeddings=True)
distances = 1 - new_x @ center
novelty = np.clip(distances / 0.35, 0, 1)

The 0.35 divisor is a starting value, not a standard. Set it from the 99th percentile of distances on benign validation data. Store that value with the model version.

Combine the signals in a transparent way:

python
risk = 0.75 * attack_probability + 0.25 * novelty

Use weights that match the environment. For agent tool calls, where indirect prompt injection is common, raise the novelty weight. For well-known fraud patterns, keep more weight on the classifier.

Choose the high threshold with a cost model

A threshold is a business decision written as code. Give each mistake a cost. Missing a real attack costs more than blocking a harmless action, but blocking every harmless action breaks the business.

Start with a simple rule: one missed attack costs as much as 20 false alarms. Then compute the cost for each candidate threshold.

python
import pandas as pd
import numpy as np

scores = pd.read_csv("scores.csv")
cost_miss = 20
cost_false_alarm = 1

rows = []
for threshold in np.arange(0.05, 0.96, 0.01):
    predicted = scores["score"] >= threshold
    misses = int(((scores["label"] == 1) & ~predicted).sum())
    false_alarms = int(((scores["label"] == 0) & predicted).sum())
    cost = misses * cost_miss + false_alarms * cost_false_alarm
    rows.append({
        "threshold": round(float(threshold), 2),
        "misses": misses,
        "false_alarms": false_alarms,
        "cost": cost,
    })

results = pd.DataFrame(rows).sort_values("cost")
print(results.head(10))

Pick the lowest cost threshold, then check two facts. First, confirm that the miss count fits the risk appetite. Second, confirm that the false alarm count fits the response team. If the cheapest threshold creates 4,000 alerts per day, it is not usable.

A bank may accept more false alarms than a small SaaS team. A production change system may accept fewer automatic blocks than a document parser. Keep the rationale next to the threshold in the repository.

Choose the low threshold from review capacity

The low threshold should follow the queue, not a feeling. Decide how many events analysts can review per day. Then pick the highest recall that fits that budget.

python
daily_events = 100000
review_capacity = 500
review_rate = review_capacity / daily_events

candidates = np.arange(0.05, 0.96, 0.01)
best = None
for threshold in candidates:
    flagged = scores["score"] >= threshold
    flagged_rate = flagged.mean()
    recall = flagged[scores["label"] == 1].mean()
    if flagged_rate <= review_rate:
        if best is None or recall > best["recall"]:
            best = {
                "threshold": round(float(threshold), 2),
                "review_rate": float(flagged_rate),
                "recall": float(recall),
            }

print(best)

This gives a review band. Events below the low threshold are logged. Events between the two thresholds go to analysts. Events above the high threshold take automatic action. If the two thresholds are close, the detector has weak separation. Improve the data before forcing the numbers apart.

Test on unseen attack families

Random splits leak patterns. A camouflaged attack from a new family needs a harder test. Group the data by attack family and hold out entire families.

python
from sklearn.model_selection import LeaveOneGroupOut

encoder = SentenceTransformer("BAAI/bge-m3")
embeddings = encoder.encode(df["text"].astype(str).tolist(), normalize_embeddings=True)
labels = df["label"].astype(int).values
groups = df["family"].fillna("benign").astype(str).values

splitter = LeaveOneGroupOut()
family_results = []

for train_idx, test_idx in splitter.split(embeddings, labels, groups):
    base = LogisticRegression(C=1.0, class_weight="balanced", max_iter=2000)
    model = CalibratedClassifierCV(base, method="sigmoid", cv=3)
    model.fit(embeddings[train_idx], labels[train_idx])
    held_out_scores = model.predict_proba(embeddings[test_idx])[:, 1]

    family = groups[test_idx][0]
    for threshold in [0.50, 0.70, 0.85]:
        predicted = held_out_scores >= threshold
        tp = int(((labels[test_idx] == 1) & predicted).sum())
        fn = int(((labels[test_idx] == 1) & ~predicted).sum())
        fp = int(((labels[test_idx] == 0) & predicted).sum())
        family_results.append({
            "family": family,
            "threshold": threshold,
            "tp": tp,
            "fn": fn,
            "fp": fp,
        })

print(pd.DataFrame(family_results).sort_values(["family", "threshold"]))

A single threshold will behave differently across families. Research on conformal prediction for network intrusion triage reported family-dependent novelty behavior, including large differences between web attacks and port scans. Treat that as a warning. One global number can hide weak coverage for the attack type that matters most.

If one family fails, do not lower the global threshold until every alert floods the queue. Add examples from that family, add a dedicated rule, or use a separate threshold for that event class.

Put the threshold into production

Store the threshold, model name, calibration value, training date, and cost assumptions in one config file.

json
{
  "model": "BAAI/bge-m3",
  "classifier": "attack_classifier-v7.joblib",
  "novelty_divisor": 0.35,
  "risk_weights": {
    "classifier": 0.75,
    "novelty": 0.25
  },
  "thresholds": {
    "review": 0.42,
    "automatic_action": 0.86
  },
  "cost_assumptions": {
    "missed_attack": 20,
    "false_alarm": 1
  }
}

At inference time, write the score and its parts to the alert record.

python
record = {
    "event_id": event_id,
    "risk": float(risk),
    "attack_probability": float(attack_probability),
    "novelty": float(novelty),
    "model_version": "v7",
    "threshold_version": "2026-11-17",
}

An analyst needs to see why the event crossed the line. A bare score produces appeal, not trust.

Guard against common failure modes

  • A threshold tuned on obvious attacks will fail on camouflage. Add hard negatives and unseen families to the test set.
  • A random split can inflate results. Split by family, tenant, time period, or actor group.
  • Calibration can drift when input text changes. Recheck it after log schema changes.
  • Novelty can flag new but harmless work. Keep novelty as one signal, not the final judge.
  • A single global threshold can hide weak coverage for rare attacks. Use class-specific thresholds when the data supports them.
  • Analysts can learn to ignore alerts that lack evidence. Store score parts and source snippets.
  • Attackers can adapt once they know the words that trigger detection. Combine semantic detection with identity, device, network, and behavior signals.

Keep the system honest

Track four numbers every week:

  • Alert volume by threshold band.
  • True positive rate by attack family.
  • False positive rate by event source.
  • Time from first score to analyst decision.

Review a sample of low-score events every week. Those events are the blind spot. If they contain real attacks, the threshold did not fail by itself. The evaluation set missed a pattern.

NIST's AI Risk Management Framework Measure function asks teams to track metrics against intended behavior and acceptable limits. That fits detection work. Write the acceptable limit before the incident, not after it.

Final assessment

An open-source model gives you a risk score. Threshold tuning turns that score into a security decision. Use BGE-M3 or another inspectable model, calibrate the output, add novelty for camouflage, and choose thresholds from cost and capacity.

The best protection threshold is the one that survives a hard evaluation set, fits the response team, and has a written rationale. Revisit it when traffic changes. Attackers adapt, and a threshold that worked last quarter is only evidence of what worked last quarter.

Sources

Caleb Ngeno

About the author

Caleb Ngeno

Lead Developer

Founder and lead developer with expertise in full-stack development and cloud solutions.

Share this article

Work with TechGaetano

Hardening a production environment?

TechGaetano helps teams design zero-trust controls, review identity and network boundaries, and turn security policy into engineering practice.

Talk to a security engineer

Frequently asked questions

What is Using an Open-Source AI Model to Detect Camouflaged Attacks?

Abstract. An open source model can help detect camouflaged attacks, but the model is only half of the system. The other half is the threshold that decides when a score becomes an alert, a quarantine, or no action. This article shows how to tune that threshold .

How should a team get started with Using an Open-Source AI Model to Detect Camouflaged Attacks?

Start with asset and identity inventory, define the trust boundaries, and roll out controls in phases: identity, device posture, least-privilege access, segmentation, then continuous verification.

What should teams avoid when working on Using an Open-Source AI Model to Detect Camouflaged Attacks?

The main risks are treating zero trust as a product purchase, leaving standing privileges in place, and assuming the model or the network perimeter is a security boundary.

Further reading

Related posts