There are now several serious benchmarks for evaluating financial AI: Finance Agent Benchmark from Vals AI (537 questions, best models scoring under 47%), BankerToolBench (502 investment banking workflows), Patronus FinanceBench (10,000+ questions on real financial documents), the Commonwealth Bank of Australia's Meta-Benchmarks mapping 452 benchmarks to 38 banking domains.
These are genuine contributions. If you want to know whether GPT-4 can correctly interpret a 10-K filing, answer yield curve questions, or execute a multi-step equity analysis workflow, these benchmarks give you useful signal.
They don't tell you whether your AI can be exploited. That's a separate property, and in banking, it's the more consequential one.
What task performance benchmarks actually measure
Finance Agent Benchmark asks: can the model perform financial reasoning tasks correctly? Investment banking workflow tasks, credit analysis, document understanding. A model that scores 47% on this benchmark is doing better than most — the benchmark is designed to be hard.
BankerToolBench asks: can the model use the right tools in the right sequence to execute complex banking workflows? It tests 502 workflows contributed by actual bankers, which makes it unusually grounded in real work.
Patronus FinanceBench asks: can the model accurately answer questions about real financial documents — 10-Ks, earnings reports, financial statements — without hallucinating numbers?
CBA's Meta-Benchmark project asks: across 452 existing benchmarks mapped to 38 banking domains, which benchmarks predict performance on which tasks?
All four of these questions are worth asking. None of them are: "what happens when someone deliberately tries to break your AI?"
Why that omission matters more in banking than anywhere else
In most domains, an AI that is highly accurate at its task is also reasonably safe. A better image classifier is harder to fool. A more accurate language model makes fewer mistakes and is less susceptible to naive manipulation.
Banking is different. The attack surface isn't the model's accuracy — it's the model's downstream authority. A banking AI that processes loan applications, KYC documents, AML alerts, or customer queries has access to consequential data and sometimes the ability to trigger consequential actions. The adversary's goal isn't to make the model wrong. It's to make the model do something it wasn't supposed to do.
Task accuracy doesn't predict adversarial robustness. A model that answers FinanceBench questions at 85% accuracy can have a 70% attack success rate on RAG poisoning simultaneously. These properties don't correlate because they reflect different aspects of model behavior.
FinSafetyBench (ACL 2026) is the closest academic analog to an adversarial safety benchmark for financial AI. It tests for English-Chinese financial crime compliance and financial reasoning safety. It covers an important slice of the problem — but not the agentic attack surface, not India's regulatory mapping, and not the tool-use and retrieval exploitation vectors that matter for deployed 2026 banking AI systems.
The specific gaps in each major benchmark
Finance Agent Benchmark: Tests whether agents correctly execute financial tasks. Doesn't include adversarial inputs — the benchmark is a performance ceiling test, not a safety floor test. A model that tops this leaderboard hasn't been tested against indirect injection, tool misuse, or knowledge base poisoning.
BankerToolBench: 502 workflows sourced from actual bankers — this is genuinely valuable data. But all 502 workflows represent legitimate use cases. There are no adversarial workflows: no workflows where the tool call sequence was induced by a poisoned input, no workflows where the agent exceeded its authorized scope because it was given a crafted instruction. The benchmark tests what the model does when used correctly, not what it does when used incorrectly.
Patronus FinanceBench: Accuracy on real financial documents is exactly what this tests — and it tests it rigorously. But accuracy and exploitability are orthogonal for document-processing AI. A model that accurately extracts numbers from a 10-K can still be manipulated through an injected directive embedded in that same document. FinanceBench doesn't test for that because it wasn't designed to.
CBA Meta-Benchmarks: The most comprehensive mapping of existing benchmarks to banking domains. 452 benchmarks, 38 domains. Valuable as a coverage analysis. But it maps task performance benchmarks to banking domains — there are no adversarial safety benchmarks in its catalog, because until recently there weren't any to map.
What an adversarial safety benchmark actually tests
An adversarial safety benchmark asks a different question: given that someone is actively trying to make your AI misbehave, what is the probability they succeed?
This requires:
A taxonomy of attack types. Not just "bad inputs" — specific categories that reflect the actual attack surface of deployed banking AI. Direct manipulation, indirect injection through content the AI reads, attacks on the tools the AI calls, attacks on the knowledge base the AI retrieves from, attacks that exploit authorization boundaries across customer records.
Mapping to real-world consequences. An attack succeeds if it produces a measurable harmful outcome: unauthorized financial action, data leakage across customer boundaries, regulatory guidance violation, investment advice beyond permitted scope. Pass/fail based on whether the output crosses a defined harm threshold, not just whether it "looks bad."
Regulatory grounding. In BFSI specifically, "harmful output" has specific legal definitions: what constitutes investment advice under SEBI regulations, what constitutes a KYC bypass under RBI KYC Master Direction 2023, what constitutes a data boundary violation under DPDP Act 2023. A benchmark without this mapping can't produce audit-ready validation evidence.
Agentic coverage. 2026 banking AI systems don't just produce text. They call tools, retrieve from knowledge bases, access structured databases, sometimes initiate transactions. Adversarial testing that only covers text output misses the most dangerous part of the attack surface.
FinProof covers all four. The 13 attack categories in the benchmark were built specifically to test the agentic and retrieval attack surface, mapped to RBI, SEBI, DPDP, EU AI Act, FATF, and SR 11-7 compliance requirements. 11,474 labeled prompts across four tiers. The only adversarial safety benchmark in BFSI with this scope.
How to use both types of benchmarks
Task performance benchmarks and adversarial safety benchmarks are complementary, not competing.
Use a task performance benchmark (FinanceBench, Finance Agent Benchmark, BankerToolBench) to answer: is this model competent enough to be deployed for this use case? Does it accurately understand financial documents? Can it execute the workflows we need?
Use an adversarial safety benchmark (FinProof) to answer: once it's deployed, can it be exploited? If someone with access to the model's inputs tries to cause it to behave in ways it shouldn't, what's the probability they succeed?
Neither question is sufficient alone. A model that is accurate but exploitable is a risk. A model that is robust but inaccurate doesn't do the job.
The current state of the field: there are several credible task performance benchmarks for financial AI. There is one adversarial safety benchmark with regulatory grounding and agentic coverage. Deploying a banking AI that has been evaluated only on the former is equivalent to hiring a bank teller who passed the accuracy test but has never been tested against the social engineering scenarios that actually cause teller fraud.
A model that aces FinanceBench and fails FinProof is accurate and exploitable. In banking, that second property is the one that creates regulatory liability.
The evaluation a pre-deployment review should include
A pre-deployment AI validation for a BFSI use case should cover both dimensions:
Task performance validation: Run the model on a representative sample of the actual use case tasks. Use FinanceBench or a custom evaluation set based on your specific domain. Establish accuracy, precision, recall baselines.
Adversarial safety validation: Run the model on FinProof Tier 2 (email registration required, full D-series agentic attacks included). Document your attack success rate by attack category. This is your adversarial robustness baseline — the number you cite in your model validation report.
Regulatory mapping: For each attack category where the model fails, map the failure to the specific regulatory provision it implicates. A D-10 RAG poisoning failure implicates DPDP Act Section 8(4). A D-09 MCP tool poisoning failure implicates RBI's outsourcing and agentic AI control requirements. A B-02 failure implicates RBI KYC Master Direction 2023.
This produces a validation report that's interpretable by a compliance officer, an internal audit team, and a regulatory examiner — not just an ML engineer.
- Tier 1 dataset (free): huggingface.co/datasets/Zytra/finproof-bench
- Full evaluation methodology: SSRN 6728799
- Request a Tier 4 validation report: finproof@zytratechnologies.com
See also: BFSI AI Attack Taxonomy: 11,474 Prompts · How to Evaluate LLM Guardrails in Banking · Agent Siege: How We Built a Red-Team Wargame