Article · Cybersecurity
Using an Open-Source AI Model to Detect Camouflaged Attacks: A Threshold-Tuning Guide
By Caleb Ngeno · November 17, 2026

Abstract. An open-source model can help detect camouflaged attacks, but the model is only half of the system. The other half is the threshold that decides when a score becomes an alert, a quarantine, or no action. This article shows how to tune that threshold with a labeled evaluation set, hard negatives, calibrated scores, and a cost model. It uses BGE-M3, an MIT-licensed embedding model, with a calibrated classifier and a small novelty score. The result is a two-level policy: a high threshold for automatic action and a low threshold for human review. The article gives code, explains how to test against attack families held out from training, and shows how to monitor drift. The main point is practical. A threshold is not universal. It is a decision rule that belongs to one environment, one risk appetite, and one review capacity.
Keywords: open-source AI model, attack detection, threshold tuning, camouflaged attacks, BGE-M3, calibrated classifier, anomaly detection, prompt injection, threat detection, security analytics
Camouflaged attacks are hard because they copy the shape of normal work. A malicious instruction hides inside a support ticket. A credential dump looks like a backup job. An indirect prompt injection sits inside a document that an agent reads. A rare but valid admin action appears beside a familiar service account.
A model can read those patterns. A threshold decides what the team does about them. Set it too low and analysts drown in alerts. Set it too high and the detector misses the attack it was built to catch.
The goal is not a magic number. The goal is a threshold you can defend with test results.
What the detector should produce
Start with the decision you want the system to make. A useful detector returns a score between 0 and 1, where a higher score means higher risk. It also returns the reason for the score, such as a matched attack pattern, a high novelty value, or both.
Use three outcomes:
- Below the low threshold: allow the action and keep the log.
- Between the low and high thresholds: send it to a human queue.
- Above the high threshold: quarantine, block, or step up authentication.
This design gives the team room to make mistakes. Automatic blocking needs strong evidence. Human review can handle softer signals.
Choose a model you can inspect
BGE-M3 is a good starting point for text-heavy detection. It is open source, MIT licensed, multilingual, and handles long context. You can run it locally, so event text does not need to leave your environment. The project publishes the model card and license on Hugging Face.
BGE-M3 does not produce a risk score by itself. It turns text into embeddings. Add a small calibrated classifier on top, then add a novelty score for events that look far from benign traffic. This keeps the system explainable enough for security work.
Qwen2.5-7B-Instruct is Apache 2.0 licensed and works well as a second-stage analyst assistant. It can summarize why an event scored high. Keep it out of the final decision path until you have evaluated it. A language model that writes a good explanation can still be a poor judge.
Build the evaluation set first
Collect event summaries from logs, alerts, tickets, agent tool calls, and identity records. Normalize each event into one text field. Keep a family label so you can hold out whole attack types during testing.
A simple record looks like this:
{
"event_id": "evt-1042",
"actor": "svc_backup",
"action": "read",
"resource": "prod/db/orders",
"text": "backup service read orders table from new subnet after token refresh",
"family": "benign",
"label": 0
}
Then add hard negatives. These are benign events that look like attacks:
- A security team running an authorized scan.
- A backup job reading many tables at night.
- A developer using a new VPN region.
- An admin changing a mailbox rule during an incident.
- A document that mentions credentials in a training guide.
Add camouflaged positives too:
- Prompt injection inside a retrieved document.
- A script renamed to match a maintenance job.
- A suspicious command wrapped in routine deployment text.
- An account takeover that follows normal working hours.
- A rare action from a service account with a familiar name.
If the evaluation set only contains obvious cases, the threshold will look excellent in a notebook and fail in production.
Turn text into calibrated scores
Install the dependencies:
pip install sentence-transformers scikit-learn pandas numpy
The following script embeds text, fits a calibrated classifier, and saves the model. It uses a stratified split for the first pass. Later, use a family-held-out split for the real test.
import joblib
import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
MODEL_NAME = "BAAI/bge-m3"
df = pd.read_json("events.jsonl", lines=True)
train_text, test_text, train_y, test_y = train_test_split(
df["text"].astype(str),
df["label"].astype(int),
test_size=0.25,
stratify=df["label"],
random_state=42,
)
encoder = SentenceTransformer(MODEL_NAME)
train_x = encoder.encode(train_text.tolist(), normalize_embeddings=True)
test_x = encoder.encode(test_text.tolist(), normalize_embeddings=True)
base = LogisticRegression(
C=1.0,
class_weight="balanced",
max_iter=2000,
)
model = CalibratedClassifierCV(base, method="sigmoid", cv=5)
model.fit(train_x, train_y)
probabilities = model.predict_proba(test_x)[:, 1]
joblib.dump(model, "attack_classifier.joblib")
pd.DataFrame({
"text": test_text.values,
"label": test_y.values,
"score": probabilities,
}).to_csv("scores.csv", index=False)
Calibration matters because a threshold only means something when the score has a consistent meaning. A raw model output of 0.80 may behave like 0.40 on another data set. A calibrated score gives the team a stable operating language.
Add a novelty signal
Camouflaged attacks often look unlike any approved example. A classifier trained on past attacks can miss them. A novelty score catches part of that gap.
Compute the average embedding of benign training events. Then measure cosine distance from each new event to that center. Calibrate the distance against benign validation data and convert it into a 0 to 1 score.
import numpy as np
from sentence_transformers import SentenceTransformer
encoder = SentenceTransformer("BAAI/bge-m3")
benign_texts = [
"backup service read orders table after token refresh",
"developer opened staging logs from approved vpn",
"admin changed mailbox rule during incident review",
]
benign_x = encoder.encode(benign_texts, normalize_embeddings=True)
center = benign_x.mean(axis=0)
center = center / np.linalg.norm(center)
new_texts = [
"maintenance job exported private keys to a new bucket",
"support agent opened attachment with hidden instruction",
]
new_x = encoder.encode(new_texts, normalize_embeddings=True)
distances = 1 - new_x @ center
novelty = np.clip(distances / 0.35, 0, 1)
The 0.35 divisor is a starting value, not a standard. Set it from the 99th percentile of distances on benign validation data. Store that value with the model version.
Combine the signals in a transparent way:
risk = 0.75 * attack_probability + 0.25 * novelty
Use weights that match the environment. For agent tool calls, where indirect prompt injection is common, raise the novelty weight. For well-known fraud patterns, keep more weight on the classifier.
Choose the high threshold with a cost model
A threshold is a business decision written as code. Give each mistake a cost. Missing a real attack costs more than blocking a harmless action, but blocking every harmless action breaks the business.
Start with a simple rule: one missed attack costs as much as 20 false alarms. Then compute the cost for each candidate threshold.
import pandas as pd
import numpy as np
scores = pd.read_csv("scores.csv")
cost_miss = 20
cost_false_alarm = 1
rows = []
for threshold in np.arange(0.05, 0.96, 0.01):
predicted = scores["score"] >= threshold
misses = int(((scores["label"] == 1) & ~predicted).sum())
false_alarms = int(((scores["label"] == 0) & predicted).sum())
cost = misses * cost_miss + false_alarms * cost_false_alarm
rows.append({
"threshold": round(float(threshold), 2),
"misses": misses,
"false_alarms": false_alarms,
"cost": cost,
})
results = pd.DataFrame(rows).sort_values("cost")
print(results.head(10))
Pick the lowest cost threshold, then check two facts. First, confirm that the miss count fits the risk appetite. Second, confirm that the false alarm count fits the response team. If the cheapest threshold creates 4,000 alerts per day, it is not usable.
A bank may accept more false alarms than a small SaaS team. A production change system may accept fewer automatic blocks than a document parser. Keep the rationale next to the threshold in the repository.
Choose the low threshold from review capacity
The low threshold should follow the queue, not a feeling. Decide how many events analysts can review per day. Then pick the highest recall that fits that budget.
daily_events = 100000
review_capacity = 500
review_rate = review_capacity / daily_events
candidates = np.arange(0.05, 0.96, 0.01)
best = None
for threshold in candidates:
flagged = scores["score"] >= threshold
flagged_rate = flagged.mean()
recall = flagged[scores["label"] == 1].mean()
if flagged_rate <= review_rate:
if best is None or recall > best["recall"]:
best = {
"threshold": round(float(threshold), 2),
"review_rate": float(flagged_rate),
"recall": float(recall),
}
print(best)
This gives a review band. Events below the low threshold are logged. Events between the two thresholds go to analysts. Events above the high threshold take automatic action. If the two thresholds are close, the detector has weak separation. Improve the data before forcing the numbers apart.
Test on unseen attack families
Random splits leak patterns. A camouflaged attack from a new family needs a harder test. Group the data by attack family and hold out entire families.
from sklearn.model_selection import LeaveOneGroupOut
encoder = SentenceTransformer("BAAI/bge-m3")
embeddings = encoder.encode(df["text"].astype(str).tolist(), normalize_embeddings=True)
labels = df["label"].astype(int).values
groups = df["family"].fillna("benign").astype(str).values
splitter = LeaveOneGroupOut()
family_results = []
for train_idx, test_idx in splitter.split(embeddings, labels, groups):
base = LogisticRegression(C=1.0, class_weight="balanced", max_iter=2000)
model = CalibratedClassifierCV(base, method="sigmoid", cv=3)
model.fit(embeddings[train_idx], labels[train_idx])
held_out_scores = model.predict_proba(embeddings[test_idx])[:, 1]
family = groups[test_idx][0]
for threshold in [0.50, 0.70, 0.85]:
predicted = held_out_scores >= threshold
tp = int(((labels[test_idx] == 1) & predicted).sum())
fn = int(((labels[test_idx] == 1) & ~predicted).sum())
fp = int(((labels[test_idx] == 0) & predicted).sum())
family_results.append({
"family": family,
"threshold": threshold,
"tp": tp,
"fn": fn,
"fp": fp,
})
print(pd.DataFrame(family_results).sort_values(["family", "threshold"]))
A single threshold will behave differently across families. Research on conformal prediction for network intrusion triage reported family-dependent novelty behavior, including large differences between web attacks and port scans. Treat that as a warning. One global number can hide weak coverage for the attack type that matters most.
If one family fails, do not lower the global threshold until every alert floods the queue. Add examples from that family, add a dedicated rule, or use a separate threshold for that event class.
Put the threshold into production
Store the threshold, model name, calibration value, training date, and cost assumptions in one config file.
{
"model": "BAAI/bge-m3",
"classifier": "attack_classifier-v7.joblib",
"novelty_divisor": 0.35,
"risk_weights": {
"classifier": 0.75,
"novelty": 0.25
},
"thresholds": {
"review": 0.42,
"automatic_action": 0.86
},
"cost_assumptions": {
"missed_attack": 20,
"false_alarm": 1
}
}
At inference time, write the score and its parts to the alert record.
record = {
"event_id": event_id,
"risk": float(risk),
"attack_probability": float(attack_probability),
"novelty": float(novelty),
"model_version": "v7",
"threshold_version": "2026-11-17",
}
An analyst needs to see why the event crossed the line. A bare score produces appeal, not trust.
Guard against common failure modes
- A threshold tuned on obvious attacks will fail on camouflage. Add hard negatives and unseen families to the test set.
- A random split can inflate results. Split by family, tenant, time period, or actor group.
- Calibration can drift when input text changes. Recheck it after log schema changes.
- Novelty can flag new but harmless work. Keep novelty as one signal, not the final judge.
- A single global threshold can hide weak coverage for rare attacks. Use class-specific thresholds when the data supports them.
- Analysts can learn to ignore alerts that lack evidence. Store score parts and source snippets.
- Attackers can adapt once they know the words that trigger detection. Combine semantic detection with identity, device, network, and behavior signals.
Keep the system honest
Track four numbers every week:
- Alert volume by threshold band.
- True positive rate by attack family.
- False positive rate by event source.
- Time from first score to analyst decision.
Review a sample of low-score events every week. Those events are the blind spot. If they contain real attacks, the threshold did not fail by itself. The evaluation set missed a pattern.
NIST's AI Risk Management Framework Measure function asks teams to track metrics against intended behavior and acceptable limits. That fits detection work. Write the acceptable limit before the incident, not after it.
Final assessment
An open-source model gives you a risk score. Threshold tuning turns that score into a security decision. Use BGE-M3 or another inspectable model, calibrate the output, add novelty for camouflage, and choose thresholds from cost and capacity.
The best protection threshold is the one that survives a hard evaluation set, fits the response team, and has a written rationale. Revisit it when traffic changes. Attackers adapt, and a threshold that worked last quarter is only evidence of what worked last quarter.
Sources

About the author
Caleb Ngeno
Lead Developer
Founder and lead developer with expertise in full-stack development and cloud solutions.
Hardening a production environment?
TechGaetano helps teams design zero-trust controls, review identity and network boundaries, and turn security policy into engineering practice.
Talk to a security engineerFrequently asked questions
What is Using an Open-Source AI Model to Detect Camouflaged Attacks?
Abstract. An open source model can help detect camouflaged attacks, but the model is only half of the system. The other half is the threshold that decides when a score becomes an alert, a quarantine, or no action. This article shows how to tune that threshold .
How should a team get started with Using an Open-Source AI Model to Detect Camouflaged Attacks?
Start with asset and identity inventory, define the trust boundaries, and roll out controls in phases: identity, device posture, least-privilege access, segmentation, then continuous verification.
What should teams avoid when working on Using an Open-Source AI Model to Detect Camouflaged Attacks?
The main risks are treating zero trust as a product purchase, leaving standing privileges in place, and assuming the model or the network perimeter is a security boundary.
Further reading
Related posts

Malicious AI Hacking Tools: A Defense Guide to WormGPT, ARTEX, and 10 More Threats
Abstract. Malicious AI tools are no longer a single chatbot with an unsafe name. Public reporting in 2025 and 2026 shows a market of paid dark LLMs, f…

AI-Automated Hacking: Google, FBI Warnings, ARTEX, and the Risk to Large Enterprises
Abstract. Automated hacking has moved from chatbot assistance to agent workflows that can scan, decide, act, and learn with little human input. Google…