Walkthrough 07: Signal Vendor Due Diligence — Not Getting Scammed

This is the last piece in the validation and debunking track. Everything before it was about how you fool yourself — purged cross-validation, the deflated Sharpe, probabilistic backtesting, all handling the bias you generate while searching for your own strategies. This one handles the case where someone else brings you the same bias on purpose. The tools are identical.

Three Months Later You Find Out What You Bought

You find a signal vendor online. The website has professional charts and a verified Myfxbook account showing a 3.2 Sharpe over 3 years of trading. The price is $200/month.

You subscribe. Three months later the signals are losing money. The Myfxbook account still shows great performance — but your trades don’t match it.

There are a few distinct ways that mismatch happens, worth separating because the defenses differ. One is the lagging signal: the historical signals they show you are excellent, but the ones they send arrive with a delay, so you always enter after the move is done. Another is account divergence: the Myfxbook is real, but they trade differently for themselves than they signal to you, and subscriber orders become the second mover. The third is the plain one, and the most common — the signals looked great in backtest and don’t work live, because they were overfitted from the start.

The first two are fraud. The third is not, and the third is most of the market. Even a completely honest vendor is probably selling a strategy overfitted to historical data; that “verified track record” may come from a live account opened specifically to look good, rather than from the logic they are actually selling. So due diligence should not be aimed at spotting crooks. It should be aimed at measuring the strength of the evidence.

You Can Compute What That Track Record Is Worth

This arithmetic is worth working through by hand, because it explains every requirement that follows.

Suppose a vendor tests 100 strategy variants. The best one has a backtest Sharpe of 2.8. They open a live account with it, it performs well for 6 months, and they market the signals on that 6-month record.

Start with the 2.8. It did not come from one strategy; it is the maximum of 100. Even if all 100 have no edge whatsoever, the expected maximum is far above zero:

where $N$ is the number of trials, $V$ is the cross-trial standard deviation of the Sharpe estimates, $\gamma \approx 0.5772$ is the Euler-Mascheroni constant, and $Z^{-1}$ is the standard normal quantile function. Under the no-edge null, $V \approx 1/\sqrt{T}$ with $T$ the number of return observations.

Plug in: at $N = 100$ the bracket is about 2.53. For 3 years of daily data, $T = 756$ and $V \approx 0.036$; the product is roughly 0.092 in daily terms, or about 1.5 annualized. So a hundred strategies with no edge at all will, on luck alone, produce a best-of-batch annualized Sharpe near 1.5. Their 2.8 does clear that noise ceiling, but by a much smaller margin than “2.8 versus 0” suggests — and that assumes they told you honestly that they ran 100 variants.

Now the 6-month live record. For a strategy with zero true edge, the standard error of the annualized Sharpe estimate is roughly $\sqrt{252/T}$. Over six months ($T \approx 126$) that comes to 1.41, so a genuinely worthless strategy posting an annualized Sharpe of 2 over half a year sits less than 1.5 standard errors from zero — entirely ordinary noise. That live verification has ruled out almost nothing.

The same expression explains the industry convention of 2-3 years: at one year the standard error is 1.0, at three years it falls to 0.58, and quadrupling the sample only halves the noise. The rule is not custom, it is arithmetic.

So your $200/month plus spreads buys a claim whose evidentiary strength is close to zero — and none of that requires the vendor to be acting in bad faith.

Eight Red Flags, Sorted by What They Actually Ask

Red flag lists get memorized as eight slogans. In practice they are asking three different kinds of question.

The first two ask whether the record exists at all. No live track record, or less than 2 years — a backtest is not a record, and by the arithmetic above six months of live trading is not a record either. You want at least 2-3 years of independently verified live performance, evidenced by broker statements rather than Myfxbook screenshots. Myfxbook can be connected to any broker account; it establishes that an account exists, not that the account is running the strategy being sold to you. Performance too good to be believed is the same question from the other side: a Sharpe above 2.0 sustained over 3+ years is rare, and 3.5 should make you very suspicious. Ask how much leverage is in there, whether the backtest window is long enough, and how many strategies were tested. That last question matters most and gets asked least.

The middle three ask what the numbers are not telling you. “Works on all pairs” or “all market conditions” — no strategy works in all conditions, and anyone claiming otherwise either misunderstands their own strategy or is lying. Ask for performance broken out by regime (trending versus range-bound), by asset class, and by year. No explanation of why it works — “algorithmic trading” is not an edge and “our proprietary AI” is not an edge. They should be able to state in one paragraph which market inefficiency the strategy exploits. “We exploit the momentum effect in FX majors during overlapping trading sessions” is credible because it can be falsified; “we use AI” is not credible because it excludes nothing. No discussion of transaction costs — a strategy trading 50 times a day, backtested at a 0.1 pip spread, produces a number with no information in it, since costs eating more than half of gross profit is routine at that frequency. Ask for a sensitivity analysis at 0.5x, 1x, and 2x expected costs.

The last three ask who is watching. Vendor sells data and signals — if they profit from selling you data and from trading the same signals, the conflict of interest is structural and they are positioned to trade ahead of your subscription. Look for separate companies for data and signals, or at minimum a disclosure of the relationship. No risk management explanation — signals without position sizing, stop-losses, or drawdown limits can blow up, and that risk is independent of whether the signals are any good. Ask for a written risk management methodology, not “we manage risk.” No third-party audit — independent verification is the gold standard, and an auditor reviewing methodology and results supplies something you cannot obtain yourself. Ask for the auditor’s name, or for methodology published somewhere peers can examine it.

Each of these is developed at length in Signal Vendor Due Diligence (Encyclopedia).

The Pass to Run Before Subscribing

Four tiers, because these do not carry equal weight.

Verification (must have):

• Live account, 3+ years, independently verifiable (broker statement, not just platform)
• Edge mechanism explained in specific, falsifiable terms
• Backtest assumptions disclosed (costs, slippage, data cleaning)

Analysis (should have):

• Sensitivity to transaction costs (0.5x, 1x, 2x)
• Performance by market regime
• Risk management methodology documented
• No conflict of interest (vendor doesn’t sell data they trade on)

Bonus (nice to have):

• Third-party audit
• Academic publication or peer review
• Open-source code or detailed whitepaper

Dealbreakers (if missing, walk away):

• Live record < 2 years
• No explanation of edge mechanism
• Performance that’s implausibly high (Sharpe > 3.0 without leverage)

What Counts as Evidence

“Independently verifiable” runs through that list. Concretely, it means one of these artifacts.

An independent broker statement: a PDF issued by the broker showing trades, P&L, and timestamps. Not a Myfxbook screenshot, which can be edited, and not a platform badge, which only certifies that an account exists.

An audit report: a PDF from a named auditing firm — not the vendor’s own marketing team — verifying three things: that the methodology described is what was actually traded, that the live results match the claimed performance, and that the data and backtest assumptions are reasonable.

Open-source code: the strategy logic published on GitHub with a reproducible backtest. This is the strongest tier, because it hands verification to anyone willing to spend the time, including you.

A peer-reviewed publication: a paper published in a journal or presented at a conference that describes and validates the strategy. It does not guarantee profitability — published strategies often decay precisely because they were published — but it means the methodology was examined by people who know the field, which none of the other artifacts provide.

Making the Decision

Once the checklist is done, five questions settle it. Is the live record real? If you cannot verify it independently, do not subscribe; there is no middle option. Is the edge mechanism credible? If they cannot say why it works, the strategy is probably overfitted. Is the performance plausible? A Sharpe above 2.5 without leverage deserves heavy suspicion, and apply the multiple-testing discount before judging. Is the risk management credible? If they cannot explain how they size positions and cap losses, the strategy can blow up regardless of signal quality. And price: $200/month is expensive if the realized Sharpe is 0.5 and cheap if it is 2.0 — price should be proportional to verified performance, not claimed performance.

The answers collapse into three paths. Any dealbreaker present, do not subscribe. Some concerns but credible overall, subscribe at a small position size and run it for 3 months, comparing actual against promised fill by fill. Fully credible with a risk-adjusted return that holds up, allocate the way you would to any strategy — including the same stop-loss and exit conditions you would give one you built yourself.

Recurring Traps

Trusting the platform’s “verified” badge. Myfxbook verification means the account exists. It does not verify that the signals come from that account, or that the strategy running in it is the one being sold.

Falling for the free trial. A vendor with an unusually good free trial may be showing you their best stretch of signals rather than their average ones. Ask for a full record that includes the bad periods; where the record has a gap, the gap is usually the answer.

Ignoring the prior. Walk in holding the prior that most signal vendors have no real edge. That is an assumption rather than a measured proportion, but it determines how strong the evidence has to be before you are persuaded. The lower the prior, the less confidence any given track record can buy you. In practice this is the operation from the arithmetic section: deflate their Sharpe by the number of variants they tested.

Looking at returns instead of risk-adjusted returns. A signal that makes 50% while drawing down 40% is not the same product as one that makes 20% with a 5% drawdown, however much easier the first is to advertise.

Not running it on paper first. Before committing real capital, run the signals on a demo account for at least a month and log the gap between actual and promised performance trade by trade. This step catches the lagging-signal problem directly. If the gap is large, investigate it instead of explaining it away.

The same pass applies well beyond signal services: before subscribing to any of them, before allocating real capital to any strategy, and when evaluating any external manager or fund. The only difference is how professionally the materials are prepared.

Due diligence is not optional. Vendors who can produce evidence meeting the standard above turn out to be startlingly rare — and finding that out is the largest single return the process pays.

Next Steps

That completes the validation and debunking track. All seven pieces are really about one thing: any result produced by searching has to have the search itself subtracted out first, and what remains is what you actually have.

To go back through the encyclopedia reference cards, start here: → Time Series CV (Encyclopedia)