{"test":"Test 8: telling a fake from an honest page that shows your logo","version":"Brandwarden AI engine 0.1.0, risk rules v0, logo match v0","method":"Test 8: 480 controlled pages from 8 stand-in brands, judged by Brandwarden AI on brands it had not seen","datasets":["480 controlled pages: 8 stand-in brands, 15 page kinds, 4 versions each, on computer and phone"],"results":{"onBrandsItNeverSaw":{"weights":"The ones Brandwarden AI ships with (unseen brands, split A)","alertsThatAreFakes":"100.0% of 74","fakesAlerted":"92.5% of 80","honestPagesShowingTheLogoAlerted":"0 of 80","fakesCalledLikelyLegitimate":"None, in any split. Fakes below the alert line were marked Not enough evidence, which sends them to a person"},"sameBrandsTheOtherWayRound":{"split":"Unseen brands, B","alertsThatAreFakes":"97.5% of 81","fakesAlerted":"98.8% of 80","honestPagesShowingTheLogoAlerted":"0.0% of 80"},"logoMatchAlone":{"honestPagesShowingTheLogoFlagged":"98.8% of 80","flagsThatAreFakes":"38.9% of 131"}},"limits":[{"title":"New kinds of honest page.","text":"On page kinds it never saw, 60.0% of 40 alerts were fakes and 75.0% of 32 fakes alerted, because all 16 similar-name businesses were flagged. So each new kind of honest page needs labelled examples first."},{"title":"A score is a ranking, not a chance.","text":"Scores put pages in the right order, but a score of 0.9 does not mean a 9 in 10 chance that the page is fake. The calibration error, which measures how far scores sit from true chances, was 0.072 to 0.096 on unseen brands."}],"bySplit":[{"What we counted":"Pages to learn from, to test on","Unseen brands, A":"240, 240","Unseen brands, B":"240, 240","Unseen brands and page kinds, A":"160, 128","Unseen brands and page kinds, B":"160, 128"},{"What we counted":"Ranking score out of 1","Unseen brands, A":"0.998","Unseen brands, B":"1.000","Unseen brands and page kinds, A":"0.930","Unseen brands and page kinds, B":"0.833"},{"What we counted":"Alerts that are fakes","Unseen brands, A":"100.0% of 74","Unseen brands, B":"97.5% of 81","Unseen brands and page kinds, A":"60.0% of 40","Unseen brands and page kinds, B":"60.0% of 40"},{"What we counted":"Fakes alerted","Unseen brands, A":"92.5% of 80","Unseen brands, B":"98.8% of 80","Unseen brands and page kinds, A":"75.0% of 32","Unseen brands and page kinds, B":"75.0% of 32"},{"What we counted":"Honest pages showing the logo alerted","Unseen brands, A":"0.0% of 80","Unseen brands, B":"0.0% of 80","Unseen brands and page kinds, A":"0.0% of 32","Unseen brands and page kinds, B":"0.0% of 32"},{"What we counted":"Honest pages called Likely legitimate","Unseen brands, A":"82.1% of 112","Unseen brands, B":"76.8% of 112","Unseen brands and page kinds, A":"66.7% of 48","Unseen brands and page kinds, B":"66.7% of 48"},{"What we counted":"Outcome as expected","Unseen brands, A":"89.2% of 240","Unseen brands, B":"88.8% of 240","Unseen brands and page kinds, A":"81.3% of 128","Unseen brands and page kinds, B":"81.3% of 128"},{"What we counted":"Calibration error (lower is better)","Unseen brands, A":"0.072","Unseen brands, B":"0.096","Unseen brands and page kinds, A":"0.211","Unseen brands and page kinds, B":"0.291"}],"againstSimpleRules":[{"Test pages":"Unseen brands, A","Rule":"Brandwarden AI risk rules v0","Fakes flagged":"92.5% of 80","Honest pages showing the logo flagged":"0.0% of 80","All honest pages flagged":"See the first table","Flags that are fakes":"100.0% of 74"},{"Test pages":"Unseen brands, A","Rule":"Logo match alone","Fakes flagged":"63.8% of 80","Honest pages showing the logo flagged":"98.8% of 80","All honest pages flagged":"71.4% of 112","Flags that are fakes":"38.9% of 131"},{"Test pages":"Unseen brands, A","Rule":"Asks for a secret alone","Fakes flagged":"100.0% of 80","Honest pages showing the logo flagged":"20.0% of 80","All honest pages flagged":"42.9% of 112","Flags that are fakes":"62.5% of 128"},{"Test pages":"Unseen brands and page kinds, A","Rule":"Brandwarden AI risk rules v0","Fakes flagged":"75.0% of 32","Honest pages showing the logo flagged":"0.0% of 32","All honest pages flagged":"See the first table","Flags that are fakes":"60.0% of 40"},{"Test pages":"Unseen brands and page kinds, A","Rule":"Logo match alone","Fakes flagged":"87.5% of 32","Honest pages showing the logo flagged":"100.0% of 32","All honest pages flagged":"70.8% of 48","Flags that are fakes":"45.2% of 62"},{"Test pages":"Unseen brands and page kinds, A","Rule":"Asks for a secret alone","Fakes flagged":"100.0% of 32","Honest pages showing the logo flagged":"0.0% of 32","All honest pages flagged":"33.3% of 48","Flags that are fakes":"66.7% of 48"}],"outcomesByPageKind":[{"Page kind":"Fake BVN and OTP update","Suspicious":16,"Likely legitimate":0,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Fake reward asking for card details","Suspicious":16,"Likely legitimate":0,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Fake login, name but no logo","Suspicious":16,"Likely legitimate":0,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Fake login, lookalike address","Suspicious":14,"Likely legitimate":0,"Not enough evidence":2,"Could not see the page":0},{"Page kind":"Fake login, neutral address","Suspicious":12,"Likely legitimate":0,"Not enough evidence":4,"Could not see the page":0},{"Page kind":"News article showing the logo","Suspicious":0,"Likely legitimate":16,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Shop checkout, “pay with” button","Suspicious":0,"Likely legitimate":16,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Customer reviews","Suspicious":0,"Likely legitimate":16,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Comparison table","Suspicious":0,"Likely legitimate":16,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"The brand's own login","Suspicious":0,"Likely legitimate":16,"Not enough evidence":0,"Could not see the page":0},{"Page kind":"Similar-name business with its own login","Suspicious":0,"Likely legitimate":8,"Not enough evidence":8,"Could not see the page":0},{"Page kind":"Another bank, same layout","Suspicious":0,"Likely legitimate":4,"Not enough evidence":12,"Could not see the page":0},{"Page kind":"Parked lookalike address","Suspicious":0,"Likely legitimate":0,"Not enough evidence":16,"Could not see the page":0},{"Page kind":"Empty lookalike page","Suspicious":0,"Likely legitimate":0,"Not enough evidence":16,"Could not see the page":0},{"Page kind":"Behind a “checking your browser” wall","Suspicious":0,"Likely legitimate":0,"Not enough evidence":0,"Could not see the page":16}],"weights":[{"Signal":"Pressure phrases","Weight":"+2.405","Effect on the score":"Raises it"},{"Signal":"Asks for a BVN, OTP, PIN or card details","Weight":"+2.179","Effect on the score":"Raises it"},{"Signal":"Asks for any secret","Weight":"+1.362","Effect on the score":"Raises it"},{"Signal":"Words common in fakes in the address","Weight":"+1.296","Effect on the score":"Raises it"},{"Signal":"Address similar to the brand's","Weight":"+0.552","Effect on the score":"Raises it"},{"Signal":"The brand's name in the address","Weight":"+0.426","Effect on the score":"Raises it"},{"Signal":"Logo similarity","Weight":"+0.286","Effect on the score":"Raises it"},{"Signal":"Free hosting platform","Weight":"+0.218","Effect on the score":"Raises it"},{"Signal":"Names the brand in full","Weight":"+0.071","Effect on the score":"Raises it"},{"Signal":"Reads like news, reviews, a comparison or a checkout","Weight":"0.000","Effect on the score":"None"},{"Signal":"Links to the brand's own site","Weight":"-0.399","Effect on the score":"Lowers it"},{"Signal":"Names other organisations","Weight":"-0.475","Effect on the score":"Lowers it"},{"Signal":"The brand's own address","Weight":"-0.614","Effect on the score":"Lowers it"}],"whatSurprisedUs":["The logo counts for little once honest pages are in the set. Honest pages show the logo too, so what tells a fake apart is what it asks for, how it pressures the visitor and where it lives. The logo still ties a page to the brand, and its box shows the analyst where the copy is.","New kinds of honest page are the real risk. Without a similar-name business among the pages it learned from, all 16 such pages were flagged and alerts that were fakes fell to 60.0%. With them among those pages, none were.","The first round of learning exposed two mistakes in our own code, now fixed: a shared word counted as naming the brand, and the alert line sat just above the highest honest page. Fixing them raised alerts that were fakes on unseen brands, split A, from 90.6% to 100.0%.","One fake login was first called Likely legitimate. It was a plain copy of a brand's login on a lookalike address: the logo (0.82), a username and a password, a link to the real help page and no pressure phrases. A floor rule now sends such a page to a person."],"oneCompleteInvestigation":{"steps":[{"title":"A stand-in brand with its logo","text":"Example Bank, added with its logo and its own web address."},{"title":"A controlled fake BVN page","text":"A fake page built by us, on a lookalike address, that asks for a BVN and an OTP."},{"title":"An analysis that said Suspicious","text":"Score 0.894 and logo match 0.974. The versions of the engine, the risk model and the logo match were recorded with it."},{"title":"A saved decision","text":"An analyst confirmed it as a fake. The decision was saved with who made it, when, and the version of the evidence."},{"title":"The queue and the reports agree","text":"Both lists show the same case, and its report cannot be sent without approval."},{"title":"An evidence package","text":"7 files, and every file matches its fingerprint (its hash)."},{"title":"Reopened unchanged","text":"Opened again later, the case and the evidence package came back exactly the same."}],"checks":["A new analysis records its versions and the signals behind it","Analysing the case again adds a row and keeps the first","The decision is saved with the analyst, the time and the evidence version","The queue and the report list agree","A report cannot be sent without approval","Every file in the package matches its fingerprint","The evidence version matches the one in the decision","The reopened case is unchanged","The reopened package is identical","The log of every action checks out from start to end"],"checksPassed":"10 of 10","score":0.894,"logoMatch":0.974}}