← Back to the demoBack to the start of the demoHome

Is this message a scam? Build, test and fix a real classifier

ITE Higher Nitec in AI Applications: Machine Learning, Applied NLP, Deep Learning, Responsible AI

In this lab you train a machine-learning model that reads a text message and says how likely it is to be a scam. Then you find out where it is wrong, fix it, and decide how it should be used safely.

The problem. Singapore Police Force figures for 2025, as reported in the news: about 37,000 scam cases and about S$913 million lost. Job scams were the third most common type, and young people are hit hard by job, e-commerce and phishing scams.

The data.

1. The UCI SMS Spam Collection: 5,574 real text messages, labelled spam or ham (not spam). Almeida, T. and Hidalgo, J. (2011), SMS Spam Collection, UCI Machine Learning Repository, DOI 10.24432/C5CC84, licence CC BY 4.0. 3,375 of its ham messages come from the NUS SMS Corpus, collected largely from students in Singapore, so you will see some Singlish.
2. A small 2026 Singapore set written for this lab and labelled synthetic: job, parcel, bank, government, investment and chat scams, plus everyday messages. Written from the patterns in public scam advisories. No real scam text was copied and no real person's message is in it.

Rule for the whole lab: never paste a friend's or a family member's real messages into a model or a website. That is their personal data (PDPA).

1. Hook: you are the model

Read the six messages below. For each one, vote: scam or not a scam? Write your votes down. At the end of the lab the model votes too.

HOOK = [
    "Eh u reaching soon? I already at the MRT liao",
    "SingPost: Your parcel could not be delivered. Update your address within 24 hours at sg-parcel-help.top",
    "Hi, part time job, earn $300 a day just liking videos on TikTok. Interested? Reply YES",
    "Your OTP for login is 482913. Do not share this code with anyone.",
    "FREE entry into our weekly prize draw! Text WIN to claim your reward now",
    "Mum, my phone broke, this is my new number. Can you transfer $500 to my friend? Urgent",
]
for number, message in enumerate(HOOK, start=1):
    print(number, message)

1 Eh u reaching soon? I already at the MRT liao
2 SingPost: Your parcel could not be delivered. Update your address within 24 hours at sg-parcel-help.top
3 Hi, part time job, earn $300 a day just liking videos on TikTok. Interested? Reply YES
4 Your OTP for login is 482913. Do not share this code with anyone.
5 FREE entry into our weekly prize draw! Text WIN to claim your reward now
6 Mum, my phone broke, this is my new number. Can you transfer $500 to my friend? Urgent

2. Load the data and look at it

We use pandas, the standard Python library for tables of data. Each row is one message: its label and its text.

import io
import os
import re
import urllib.request
import zipfile

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

UCI_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
SG_URL = "https://d2nke3wqqm57v5.cloudfront.net/clf/scam/data/sg-2026-synthetic.csv"
# On the lab machine the files are already here; in Colab they are downloaded.
LOCAL_UCI = "../ml/data/raw/SMSSpamCollection"
LOCAL_SG = "../web/clf/scam/data/sg-2026-synthetic.csv"

def load_uci():
    if os.path.exists(LOCAL_UCI):
        source = open(LOCAL_UCI, encoding="utf-8").read()
    else:
        archive = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(UCI_URL).read()))
        source = archive.read("SMSSpamCollection").decode("utf-8")
    rows = [line.split("\t", 1) for line in source.splitlines() if line.strip()]
    return pd.DataFrame(rows, columns=["label", "text"])

uci = load_uci()
sg = pd.read_csv(LOCAL_SG if os.path.exists(LOCAL_SG) else SG_URL)
print(uci.shape, sg.shape)
uci.head(8)

(5574, 2) (118, 5)

	label	text
0	ham	Go until jurong point, crazy.. Available only ...
1	ham	Ok lar... Joking wif u oni...
2	spam	Free entry in 2 a wkly comp to win FA Cup fina...
3	ham	U dun say so early hor... U c already then say...
4	ham	Nah I don't think he goes to usf, he lives aro...
5	spam	FreeMsg Hey there darling it's been 3 week's n...
6	ham	Even my brother is not like to speak with me. ...
7	ham	As per your request 'Melle Melle (Oru Minnamin...

uci["label"].value_counts()

label
ham     4827
spam     747
Name: count, dtype: int64

3. Clean the data

Real data is messy. Three problems to fix before a model can learn from it:

1. Duplicates. Some messages appear more than once. If a copy lands in the training set and another copy in the test set, the test is too easy: the model has already seen the answer.
2. Case. "FREE" and "free" should count as the same word.
3. Links, money and numbers. Every link is different, so the model would never learn "a link is suspicious". We replace each link with the word urltoken, each amount of money with moneytoken, and every other number with numtoken. Now the model can learn the pattern.

before = len(uci)
uci = uci.drop_duplicates(subset="text").reset_index(drop=True)
print(f"Removed {before - len(uci)} duplicate messages; {len(uci)} left")

Removed 403 duplicate messages; 5171 left

URL_PATTERN = r"https?://\S+|www\.\S+|\b[a-z0-9-]+\.(?:com|sg|net|org|top|xyz|info|link|ly|me|co|uk)\b\S*"
MONEY_PATTERN = r"[$]\s?\d[\d,]*(?:\.\d+)?"
NUMBER_PATTERN = r"\d+"

def clean(text):
    text = text.replace("\u00a3", "$").replace("\u20ac", "$")   # pounds and euros count as money too
    text = re.sub(r"[^\x20-\x7e]", " ", text)                  # keep plain characters only
    text = text.lower()
    text = re.sub(URL_PATTERN, " urltoken ", text)
    text = re.sub(MONEY_PATTERN, " moneytoken ", text)
    text = re.sub(NUMBER_PATTERN, " numtoken ", text)
    return re.sub(r"[^a-z]+", " ", text).strip()                 # drop punctuation, squeeze spaces

print(clean("WINNER!! Claim your £900 prize at www.win-now.com before 12pm"))
print(clean("Hi, part time job, earn $300 a day. Visit jobs-sg.top"))

winner claim your moneytoken prize at urltoken before numtoken pm
hi part time job earn moneytoken a day visit urltoken

4. Explore the data (EDA)

Before training, look. Three questions: how many of each class? Are scam messages longer? Which words are common in each?

uci["y"] = (uci["label"] == "spam").astype(int)
sg["y"] = (sg["label"] == "scam").astype(int)
share = uci["y"].mean()
print(f"Spam is {share:.1%} of the messages. Not spam is {1 - share:.1%}.")

uci["length"] = uci["text"].str.len()
uci.groupby("label")["length"].describe().round(0)

Spam is 12.6% of the messages. Not spam is 87.4%.

	count	mean	std	min	25%	50%	75%	max
label
ham	4518.0	71.0	57.0	2.0	34.0	53.0	91.0	910.0
spam	653.0	138.0	30.0	13.0	132.0	148.0	157.0	223.0

fig, axes = plt.subplots(1, 2, figsize=(11, 3.4))
uci["label"].value_counts().plot.bar(ax=axes[0], color=["#4c78a8", "#e45756"])
axes[0].set_title("Messages per class")
for label, colour in (("ham", "#4c78a8"), ("spam", "#e45756")):
    uci.loc[uci["label"] == label, "length"].clip(upper=300).plot.hist(ax=axes[1], bins=30, alpha=0.6, color=colour, label=label)
axes[1].set_title("Message length (characters)")
axes[1].legend()
plt.tight_layout()
plt.show()

from collections import Counter

def top_words(texts, n=12):
    counts = Counter(word for text in texts for word in clean(text).split() if len(word) > 2)
    return counts.most_common(n)

print("Spam:", top_words(uci.loc[uci.y == 1, "text"]))
print("Ham: ", top_words(uci.loc[uci.y == 0, "text"]))

Spam: [('numtoken', 2457), ('call', 322), ('you', 268), ('moneytoken', 268), ('your', 242), ('free', 197), ('for', 185), ('the', 183), ('now', 168), ('txt', 136), ('urltoken', 124), ('from', 123)]
Ham:  [('you', 1880), ('numtoken', 1105), ('the', 1066), ('and', 826), ('that', 547), ('for', 481), ('have', 426), ('can', 421), ('but', 420), ('are', 398), ('not', 393), ('your', 381)]

The accuracy trap. About 87% of messages are not spam. A "model" that always says not a scam is right 87% of the time and catches zero scams. So accuracy alone tells you almost nothing here. Keep this number in mind.

5. Features: turning words into numbers

A model cannot read. It needs numbers. Two ideas:

1. Bag of words. Count how often each word appears. Word order is thrown away.
2. TF-IDF. Like a count, but a word that appears in almost every message ("the", "you") gets a small weight, and a rarer, telling word ("claim", "urltoken") gets a bigger one.

We also count pairs of words ("prize draw", "part time"), because pairs carry meaning single words miss.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

train, test = train_test_split(uci, test_size=0.2, stratify=uci["y"], random_state=42)
print(f"Train: {len(train)} messages. Test: {len(test)} messages, kept aside and never trained on.")

def make_vectorizer():
    return TfidfVectorizer(preprocessor=clean, token_pattern=r"[a-z]+", ngram_range=(1, 2), min_df=2, sublinear_tf=True)

vectorizer = make_vectorizer()
X_train = vectorizer.fit_transform(train["text"])
X_test = vectorizer.transform(test["text"])
print(f"Each message is now a row of {X_train.shape[1]} numbers, almost all of them zero.")

Train: 4136 messages. Test: 1035 messages, kept aside and never trained on.
Each message is now a row of 10727 numbers, almost all of them zero.

example = "Claim your FREE prize now"
row = vectorizer.transform([example])
weights = {vectorizer.get_feature_names_out()[i]: round(row[0, i], 3) for i in row.nonzero()[1]}
print(clean(example), "->", weights)

claim your free prize now -> {'claim': np.float64(0.354), 'claim your': np.float64(0.505), 'free': np.float64(0.296), 'now': np.float64(0.241), 'prize': np.float64(0.367), 'your': np.float64(0.228), 'your free': np.float64(0.537)}

6. Train two models, and beat the baseline

1. Naive Bayes: counts how often each word appears in spam and in ham, and combines the evidence. Fast and a classic for text.
2. Logistic regression: learns one weight per word or pair. Positive weights push towards scam. The weights are easy to explain, which is why the web demo uses this one.

from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix

def measure(name, y_true, y_pred):
    return {
        "model": name,
        "accuracy": accuracy_score(y_true, y_pred),
        "precision": precision_score(y_true, y_pred, zero_division=0),
        "recall": recall_score(y_true, y_pred, zero_division=0),
        "f1": f1_score(y_true, y_pred, zero_division=0),
    }

baseline = measure("Always 'not a scam'", test["y"], np.zeros(len(test), dtype=int))
nb = MultinomialNB(alpha=0.1).fit(X_train, train["y"])
lr = LogisticRegression(C=10, max_iter=3000).fit(X_train, train["y"])
THRESHOLD = 0.5
p_test = lr.predict_proba(X_test)[:, 1]

results = pd.DataFrame([
    baseline,
    measure("Naive Bayes", test["y"], nb.predict(X_test)),
    measure("Logistic regression", test["y"], (p_test >= THRESHOLD).astype(int)),
]).set_index("model").round(3)
results

	accuracy	precision	recall	f1
model
Always 'not a scam'	0.873	0.000	0.000	0.000
Naive Bayes	0.989	0.984	0.931	0.957
Logistic regression	0.986	1.000	0.893	0.944

Look at the baseline row: high accuracy, zero recall. That is the accuracy trap in one line.

7. Evaluate properly: precision, recall and the threshold

1. Recall: of all the real scams, how many did we catch?
2. Precision: of everything we flagged, how many really were scams?
3. F1: one number that balances the two.

The model gives a score between 0 and 1. We choose the threshold above which we say "scam". Lower it and you catch more scams but raise more false alarms. Which mistake is worse for a scam checker: a missed scam, or a false alarm on your mum's message?

matrix = confusion_matrix(test["y"], (p_test >= THRESHOLD).astype(int))
print("                 predicted ham  predicted scam")
print(f"actually ham     {matrix[0, 0]:13d}  {matrix[0, 1]:14d}")
print(f"actually scam    {matrix[1, 0]:13d}  {matrix[1, 1]:14d}")

                 predicted ham  predicted scam
actually ham               904               0
actually scam               14             117

sweep = []
for threshold in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]:
    predicted = (p_test >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(test["y"], predicted).ravel()
    sweep.append({"threshold": threshold, "scams caught": tp, "scams missed": fn, "false alarms": fp,
                  "precision": round(precision_score(test["y"], predicted, zero_division=0), 3),
                  "recall": round(recall_score(test["y"], predicted), 3)})
sweep = pd.DataFrame(sweep)
sweep

	threshold	scams caught	scams missed	false alarms	precision	recall
0	0.1	127	4	14	0.901	0.969
1	0.2	125	6	4	0.969	0.954
2	0.3	123	8	2	0.984	0.939
3	0.4	120	11	1	0.992	0.916
4	0.5	117	14	0	1.000	0.893
5	0.6	116	15	0	1.000	0.885
6	0.7	113	18	0	1.000	0.863
7	0.8	110	21	0	1.000	0.840
8	0.9	92	39	0	1.000	0.702

plt.figure(figsize=(6, 3.4))
plt.plot(sweep["threshold"], sweep["precision"], marker="o", label="precision")
plt.plot(sweep["threshold"], sweep["recall"], marker="o", label="recall")
plt.xlabel("threshold")
plt.title("Move the threshold: catch more, or alarm less")
plt.legend()
plt.grid(alpha=0.3)
plt.show()

8. A small neural network, and overfitting

A multi-layer perceptron (MLP) is a small neural network: the TF-IDF numbers go in, pass through a hidden layer of 32 "neurons", and a scam score comes out. It can learn combinations of words that logistic regression cannot.

Watch two scores: on the training data and on the test data. A big gap means the network has memorised the training messages instead of learning the pattern. That is overfitting.

from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(hidden_layer_sizes=(32,), max_iter=40, random_state=42)
mlp.fit(X_train, train["y"])
mlp_train = measure("MLP on training data", train["y"], mlp.predict(X_train))
mlp_test = measure("MLP on test data", test["y"], mlp.predict(X_test))
pd.DataFrame([mlp_train, mlp_test]).set_index("model").round(3)

/Users/sri/Documents/workspace/AWS AI agents/AWS Agentic AI App/prototypes/ite-classifier/.venv/lib/python3.13/site-packages/sklearn/neural_network/_multilayer_perceptron.py:785: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (40) reached and the optimization hasn't converged yet.
  warnings.warn(

	accuracy	precision	recall	f1
model
MLP on training data	1.000	1.000	1.000	1.000
MLP on test data	0.988	0.992	0.916	0.952

plt.figure(figsize=(6, 3))
plt.plot(mlp.loss_curve_)
plt.xlabel("training round (epoch)")
plt.ylabel("loss (error)")
plt.title("The network's error on the training data falls towards zero")
plt.grid(alpha=0.3)
plt.show()

Training recall is perfect; test recall is lower. The network is not better than logistic regression here, and it is much harder to explain. For a tool that must say why, the simpler model wins.

9. Found wrong: new scams, old data

The UCI data was collected around 2011 and 2012. Scams in Singapore in 2026 look different: job scams, fake parcel redelivery, bank and government impersonation, investment groups, chat lures.

We test the model on the 2026 synthetic set (the half marked test). It has never seen any of it.

sg_retrain = sg[sg["split"] == "retrain"]
sg_test = sg[sg["split"] == "test"]

p_sg_old = lr.predict_proba(vectorizer.transform(sg_test["text"]))[:, 1]
drift = pd.DataFrame([
    measure("2012 test set", test["y"], (p_test >= THRESHOLD).astype(int)),
    measure("2026 Singapore set", sg_test["y"], (p_sg_old >= THRESHOLD).astype(int)),
]).set_index("model").round(3)
drift

	accuracy	precision	recall	f1
model
2012 test set	0.986	1.0	0.893	0.944
2026 Singapore set	0.661	1.0	0.459	0.630

by_type = sg_test.assign(caught=(p_sg_old >= THRESHOLD)).query("y == 1").groupby("category")["caught"].mean().round(2)
by_type.sort_values().plot.barh(figsize=(6, 3), color="#e45756", title="Share of 2026 scams caught, by type (old model)")
plt.xlim(0, 1)
plt.show()
by_type

category
bank          0.83
chat_lure     0.17
government    0.50
investment    0.33
job           0.43
parcel        0.50
Name: caught, dtype: float64

Recall drops. Nothing in the code changed; the world did. This is called data drift, and it is why a model is never "finished".

10. Fix it: add new examples, retrain, re-measure

We add the other half of the 2026 set (marked retrain) to the training data, train again, and measure on the same 2026 test half and on the same 2012 test set. Never retrain on the test messages: then the test would be meaningless.

train_v2 = pd.concat([train[["text", "y"]], sg_retrain[["text", "y"]]], ignore_index=True)
vectorizer_v2 = make_vectorizer()
lr_v2 = LogisticRegression(C=10, max_iter=3000).fit(vectorizer_v2.fit_transform(train_v2["text"]), train_v2["y"])

p_test_v2 = lr_v2.predict_proba(vectorizer_v2.transform(test["text"]))[:, 1]
p_sg_v2 = lr_v2.predict_proba(vectorizer_v2.transform(sg_test["text"]))[:, 1]
before_after = pd.DataFrame([
    {"set": "2012 test set", "model": "old", **measure("", test["y"], (p_test >= THRESHOLD).astype(int))},
    {"set": "2012 test set", "model": "retrained", **measure("", test["y"], (p_test_v2 >= THRESHOLD).astype(int))},
    {"set": "2026 Singapore set", "model": "old", **measure("", sg_test["y"], (p_sg_old >= THRESHOLD).astype(int))},
    {"set": "2026 Singapore set", "model": "retrained", **measure("", sg_test["y"], (p_sg_v2 >= THRESHOLD).astype(int))},
]).drop(columns="model").set_index(["set"]).round(3)
before_after.insert(0, "model", ["old", "retrained", "old", "retrained"])
before_after

	model	accuracy	precision	recall	f1
set
2012 test set	old	0.986	1.0	0.893	0.944
2012 test set	retrained	0.986	1.0	0.893	0.944
2026 Singapore set	old	0.661	1.0	0.459	0.630
2026 Singapore set	retrained	0.814	1.0	0.703	0.825

missed = sg_test.assign(score=p_sg_v2.round(2)).query("y == 1 and score < @THRESHOLD")[["category", "score", "text"]]
print(f"Still missed after retraining: {len(missed)} of {int(sg_test['y'].sum())} scams")
missed

Still missed after retraining: 11 of 37 scams

	category	score	text
9	job	0.22	Hi, my company is looking for online promoters...
13	job	0.31	We are recruiting! Work 1 hour a day reviewing...
49	government	0.24	ICA customs: a parcel under your name contains...
51	investment	0.40	Hi, my uncle works at an investment bank and s...
55	investment	0.02	Hey, long time no see! I've been making good m...
57	investment	0.28	Our fund manager is giving free stock picks to...
59	investment	0.20	Hi, I got your number from the investment semi...
63	chat_lure	0.00	Hey it's me, I changed my number. Can you help...
67	chat_lure	0.02	Hello dear, I saw your profile. I am a doctor ...
71	chat_lure	0.21	Hi, this is your friend from secondary school!...
73	chat_lure	0.04	Selling the iPhone you asked about on Carousel...

Better, not perfect. The scams it still misses are mostly friendly chat lures with no link and no money words. A word-counting model cannot see intent. That is where a human, or a language model as a second opinion, helps. And next month there will be a new kind of scam: monitoring never stops.

11. Explain: which words raised the score?

Logistic regression is a sum: each word's TF-IDF value times its weight. So we can show exactly which words pushed a message towards scam. Now let the model vote on the six hook messages.

terms = vectorizer_v2.get_feature_names_out()
weights = lr_v2.coef_[0]

def explain(message, top=5):
    row = vectorizer_v2.transform([message])
    parts = sorted(((terms[i], row[0, i] * weights[i]) for i in row.nonzero()[1]), key=lambda item: -item[1])
    score = lr_v2.predict_proba(row)[0, 1]
    raised = ", ".join(f"{term} (+{value:.2f})" for term, value in parts[:top] if value > 0)
    return score, raised

for number, message in enumerate(HOOK, start=1):
    score, raised = explain(message)
    verdict = "scam" if score >= THRESHOLD else "not a scam"
    print(f"{number}. {score:.0%} -> {verdict}. Raised by: {raised or 'nothing'}")

1. 0% -> not a scam. Raised by: the (+0.08)
2. 92% -> scam. Raised by: urltoken (+1.64), your (+0.98), at urltoken (+0.89), numtoken (+0.79), parcel (+0.40)
3. 70% -> scam. Raised by: moneytoken (+1.45), reply (+0.97), earn (+0.41), a (+0.32), part time (+0.27)
4. 13% -> not a scam. Raised by: numtoken (+0.98), your (+0.72), this (+0.54), with (+0.51), for (+0.35)
5. 91% -> scam. Raised by: text (+0.98), free (+0.85), claim (+0.59), your (+0.59), our (+0.49)
6. 77% -> scam. Raised by: moneytoken (+1.42), new (+0.65), transfer (+0.60), this (+0.46), urgent (+0.44)

order = np.argsort(weights)
print("Words that push towards SCAM:     ", ", ".join(terms[order[-15:]][::-1]))
print("Words that push towards NOT SCAM: ", ", ".join(terms[order[:15]]))

Words that push towards SCAM:      urltoken, moneytoken, numtoken, call numtoken, text, free, reply, to numtoken, numtoken p, p, txt, your, call, mobile, at urltoken
Words that push towards NOT SCAM:  i, lt, gt, my, lt gt, ok, then, that, me, its, got, hey, he, lor, come

12. Responsible AI: who gets the false alarms?

A false alarm is not harmless. If a scam checker keeps flagging the way your group writes, you stop trusting it, or worse, it is used to block you.

We check the false-positive rate (real messages wrongly flagged) for messages that look Singlish and for the rest. The Singlish test below is a rough word list, not a real language detector, and the groups are small. Say that whenever you report it.

SINGLISH = re.compile(r"\b(lah|lor|leh|meh|sia|liao|hor|lei|mah|wah|walao|shiok|paiseh|bo|nvm|dun|chope|belanja)\b", re.IGNORECASE)
ham_test = pd.concat([test.loc[test.y == 0, ["text"]], sg_test.loc[sg_test.y == 0, ["text"]]], ignore_index=True)
ham_test["singlish"] = ham_test["text"].str.contains(SINGLISH)
ham_test["flagged"] = lr_v2.predict_proba(vectorizer_v2.transform(ham_test["text"]))[:, 1] >= THRESHOLD
fairness = ham_test.groupby("singlish")["flagged"].agg(["count", "sum", "mean"]).rename(
    index={True: "looks Singlish", False: "other"}, columns={"count": "real messages", "sum": "false alarms", "mean": "false-positive rate"})
fairness.round(4)

/var/folders/_f/kmb_5vpx1p131p_yzbft3pwc0000gn/T/ipykernel_84926/2528943698.py:3: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  ham_test["singlish"] = ham_test["text"].str.contains(SINGLISH)

	real messages	false alarms	false-positive rate
singlish
other	875	0	0.0
looks Singlish	51	0	0.0

How this model should be used.

1. Warn, never delete. A false alarm that deletes a real message does real harm. The demo only ever says "be careful".
2. A person decides. The score is advice. Check with the official ScamShield app or the ScamShield Helpline, 1799.
3. Protect personal data (PDPA). Do not paste friends' real messages. The web demo refuses NRIC, phone and card numbers and stores nothing you paste.
4. Be transparent. Publish what the model was trained on, how it was measured, and where it fails: a model card.
5. These match Singapore's AI Verify principles (transparency, robustness, fairness, human oversight) and the IMDA Model AI Governance Framework (human involvement, operations management, communication).

13. Assessment: three short tasks

1. Threshold. Set THRESHOLD = 0.3 and re-run sections 7 and 9. Report precision and recall on both test sets. In two sentences: is 0.3 a better choice for a scam warning tool? Why?
2. New scam type. Write five new synthetic scam messages of a type the model has not seen (for example, a fake e-commerce refund). Score them with explain(). Add them to train_v2, retrain, and score them again. What changed, and why is this test now unfair?
3. Model card. Write one paragraph: what the model was trained on, one number from your measurements, one place it fails, and one fairness risk.

Marking guide (teacher version only)

1. Threshold (4 marks). 1 mark each for correct precision and recall on each set (2). 1 mark for stating that recall rises and precision falls. 1 mark for a justified choice: for a warning tool a missed scam usually costs more than a false alarm, so a lower threshold is defensible, provided false alarms only warn and never delete.
2. New scam type (4 marks). 1 mark for five plausible, clearly synthetic messages with no real personal data. 1 mark for before scores. 1 mark for after scores showing a rise. 1 mark for recognising the after test is unfair because the model has now trained on those exact messages; a fair test needs new, unseen examples.
3. Model card (4 marks). 1 mark each: data source with its date or licence; a correct figure from their own run; a specific failure (for example chat lures with no link); a fairness risk tied to a group (for example Singlish false alarms, with the small-sample caveat).

Appendix: export the model for the web demo

This runs only on the lab machine, where ../scam/model/ exists. It writes both models (old and retrained) as plain JSON, every measured figure, and the parity cases the web demo's own tests use to prove its JavaScript scorer gives exactly the same score as this notebook. In Colab it does nothing.

import json
import sklearn

EXPORT_DIR = "../scam/model"
UNSURE_BAND = (0.3, 0.7)
CALIBRATION_EDGES = [0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0001]

def calibration(scores, labels):
    bins = []
    for low, high in zip(CALIBRATION_EDGES[:-1], CALIBRATION_EDGES[1:]):
        inside = (scores >= low) & (scores < high)
        count = int(inside.sum())
        bins.append({"from": low, "to": min(high, 1.0), "messages": count,
                     "scamShare": float(labels[inside].mean()) if count else None})
    return bins

def export_model(model_id, label, vec, model, trained_on, cal_scores, cal_labels):
    names = vec.get_feature_names_out()
    return {
        "format": "tfidf-logistic-regression/1",
        "modelId": model_id,
        "label": label,
        "trainedOn": trained_on,
        "sklearnVersion": sklearn.__version__,
        "preprocess": {
            "poundAndEuroAsMoney": True,
            "dropNonAscii": True,
            "lowercase": True,
            "urlPattern": URL_PATTERN, "urlToken": "urltoken",
            "moneyPattern": MONEY_PATTERN, "moneyToken": "moneytoken",
            "numberPattern": NUMBER_PATTERN, "numberToken": "numtoken",
            "tokenPattern": "[a-z]+",
            "ngramRange": list(vec.ngram_range),
            "sublinearTf": vec.sublinear_tf,
            "norm": vec.norm,
        },
        "vocabulary": names.tolist(),
        "idf": vec.idf_.tolist(),
        "coef": model.coef_[0].tolist(),
        "intercept": float(model.intercept_[0]),
        "threshold": THRESHOLD,
        "unsureBand": list(UNSURE_BAND),
        "calibration": calibration(cal_scores, cal_labels),
    }

def figures(y_true, scores):
    predicted = (scores >= THRESHOLD).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, predicted, labels=[0, 1]).ravel()
    return {**{k: round(float(v), 4) for k, v in measure("", y_true, predicted).items() if k != "model"},
            "messages": int(len(y_true)), "scams": int(np.sum(y_true)),
            "caught": int(tp), "missed": int(fn), "falseAlarms": int(fp), "correctlyClear": int(tn)}

if os.path.isdir(EXPORT_DIR):
    test_y, sg_y = test["y"].to_numpy(), sg_test["y"].to_numpy()
    old = export_model("scam-2012", "Trained on the 2012 UCI data only", vectorizer, lr,
                       f"UCI SMS Spam Collection, {len(train)} training messages", p_test, test_y)
    new = export_model("scam-2026", "Retrained with the 2026 Singapore examples", vectorizer_v2, lr_v2,
                       f"UCI SMS Spam Collection, {len(train)} training messages, plus {len(sg_retrain)} synthetic 2026 Singapore messages",
                       np.concatenate([p_test_v2, p_sg_v2]), np.concatenate([test_y, sg_y]))
    metrics = {
        "data": {"uciMessages": int(before), "duplicatesRemoved": int(before - len(uci)), "uciUnique": int(len(uci)),
                 "spamShare": round(float(share), 4), "train": int(len(train)), "test": int(len(test)),
                 "sgRetrain": int(len(sg_retrain)), "sgTest": int(len(sg_test)),
                 "sgTestScams": int(sg_y.sum()), "sgTestHam": int(len(sg_y) - sg_y.sum())},
        "threshold": THRESHOLD,
        "unsureBand": list(UNSURE_BAND),
        "baseline": {k: round(float(v), 4) for k, v in baseline.items() if k != "model"},
        "naiveBayes": {k: round(float(v), 4) for k, v in measure("", test["y"], nb.predict(X_test)).items() if k != "model"},
        "mlp": {"train": {k: round(float(v), 4) for k, v in mlp_train.items() if k != "model"},
                "test": {k: round(float(v), 4) for k, v in mlp_test.items() if k != "model"}},
        "old": {"uciTest": figures(test_y, p_test), "sgTest": figures(sg_y, p_sg_old)},
        "new": {"uciTest": figures(test_y, p_test_v2), "sgTest": figures(sg_y, p_sg_v2)},
        "sgCaughtByType": {
            "old": sg_test.assign(c=p_sg_old >= THRESHOLD).query("y == 1").groupby("category")["c"].mean().round(3).to_dict(),
            "new": sg_test.assign(c=p_sg_v2 >= THRESHOLD).query("y == 1").groupby("category")["c"].mean().round(3).to_dict()},
        "stillMissed": [{"category": r.category, "score": round(float(r.score), 3), "text": r.text} for r in missed.itertuples()],
        "fairness": [{"group": name, "realMessages": int(row["real messages"]), "falseAlarms": int(row["false alarms"]),
                      "falsePositiveRate": round(float(row["false-positive rate"]), 4)} for name, row in fairness.iterrows()],
        "thresholdSweep": sweep.to_dict(orient="records"),
        "topScamTerms": terms[order[-15:]][::-1].tolist(),
        "topSafeTerms": terms[order[:15]].tolist(),
    }
    cases = [{"id": f"uci-test-{i}", "text": t} for i, t in zip(test.index, test["text"])] + \
            [{"id": r.id, "text": r.text} for r in sg.itertuples()]
    texts = [c["text"] for c in cases]
    s_old = lr.predict_proba(vectorizer.transform(texts))[:, 1]
    s_new = lr_v2.predict_proba(vectorizer_v2.transform(texts))[:, 1]
    for case, a, b in zip(cases, s_old, s_new):
        case.update({"cleaned": clean(case["text"]), "scoreOld": float(a), "scoreNew": float(b)})
    parity = {"note": "Python scores for every test message, for the Node parity test. Texts: UCI SMS Spam Collection (Almeida and Hidalgo 2011, CC BY 4.0, DOI 10.24432/C5CC84) and the synthetic 2026 Singapore set.",
              "cases": cases}
    for name, payload in (("scam-2012.json", old), ("scam-2026.json", new), ("metrics.json", metrics), ("parity-cases.json", parity)):
        with open(os.path.join(EXPORT_DIR, name), "w", encoding="utf-8") as handle:
            json.dump(payload, handle, ensure_ascii=False, separators=(",", ":"))
    print(f"Exported to {EXPORT_DIR}: {len(old['vocabulary'])} and {len(new['vocabulary'])} terms, {len(cases)} parity cases")
else:
    print("Not on the lab machine: nothing exported.")

Exported to ../scam/model: 10727 and 10876 terms, 1153 parity cases