Methodology

Every number, finding and status in a Quantgrounds report comes from the fixed rules on this page. A deterministic strategy run with the same settings always gets the same numbers.

What a run does

Security pre-check. The code is parsed, not run, and checked against an import and syntax allowlist. Code that fails is rejected and never executed.
Static code review. The parsed code is checked for known look-ahead patterns and other issues (see Other checks).
Baseline backtest. The strategy runs once over the full history in the sandbox. The engine turns its target weights into positions, costs and returns.
Experiments. The strategy runs again for each parameter change, on each held-out asset, for the look-ahead test and for a determinism check. Cost, execution-delay, regime, out-of-sample and resampling results are computed from the baseline positions and returns.
Findings, statuses, verdict and score are derived from those results by the rules below.

Data

Two fixed data sets, so results are reproducible. Random shards (the default score) draw from the Ken French Data Library: the whole US market and 49 industry portfolios, daily from 1926, value-weighted over every US stock including delisted ones (no survivorship bias). "Your setup" uses adjusted daily closes for SPY (1993–2019) and 20 US stocks (to 2018), a small survivor set for picking specific tickers. Returns are daily simple returns; a year is 252 trading days.

Execution model

Strategy interfaceon_bar(ctx, data) is called once per bar and receives prices up to and including that bar. It returns target weights per asset, or None to trade nothing and let positions drift with prices. An optional prepare(prices, params) computes indicators over the whole price table at once.
FillsA decision made at the close of day t is filled at the close of day t+1. A decision never earns the return of the bar it was made on.
CostsA flat cost per side (5 basis points by default) is charged on traded notional, that is, on every change in position, including rebalancing back to a target after prices move.
ExposureLong and short weights are allowed. If the absolute weights sum to more than 1, they are scaled down to 1 and the report notes it.
AccountingDone entirely by the engine, outside the strategy's process. Strategy code never touches positions, costs or returns.
MetricsSharpe and Sortino ratios use a zero risk-free rate. A trade opens when an asset's weight leaves zero and closes when it returns to zero or changes sign.

The twelve tests

Each test below lists its method and the rule that produces a finding. Severities are critical, high, medium and low.

Test 1

Out-of-sample periods

You declare the development period: the dates you used to design and tune the strategy. One continuous backtest is split into the development period and everything before and after it. A slice needs at least 126 trading days to be reported.

When it raises a finding

Finding when the in-sample Sharpe is above 0.3 and the out-of-sample Sharpe is below half of it. High if it is at or below a quarter of it, or if the period after development has a Sharpe at or below zero; otherwise medium.
If no development period is declared, a medium finding says the test could not separate the two.

Test 2

Parameter sensitivity

Each numeric value in PARAMS is multiplied by 0.8, 0.9, 1.1 and 1.2 while the others stay fixed, and the strategy is re-run. Integers are rounded; if rounding lands on the original value, the nearest different integer is used.

When it raises a finding

Finding ("parameter cliff") when a ±10% change keeps less than 50% of the baseline Sharpe. High below 25%. The test needs a baseline Sharpe above 0.05.

Test 3

Transaction costs

The baseline positions are re-costed at 0x, 1x, 2x, 3x and 5x the cost you set. The break-even cost is the per-side cost at which net CAGR reaches zero, found by bisection between 0 and 1,000 basis points.

When it raises a finding

High when there is no positive return even before costs, or when the break-even cost is below 2x your cost. Medium below 3x.

Test 4

Execution delay

The baseline decisions are shifted one trading day later, so each is filled two closes after the signal instead of one, and the result is re-costed.

When it raises a finding

Finding when the delayed Sharpe is less than half the baseline Sharpe. High if the delayed Sharpe is at or below zero. The test needs a baseline Sharpe above 0.05.

Test 5

Unseen assets

The same code and parameters run on each declared asset alone and on up to 10 held-out assets with at least three years of data, preferring assets of the same kind (stock or ETF) as the ones you declared.

When it raises a finding

Finding when the declared assets' median Sharpe is above 0.2 and either the held-out median Sharpe is below half of it or fewer than half of the held-out assets are profitable. High if fewer than 30% are profitable.
A medium finding is also raised if the strategy raises errors on held-out assets.

Test 6

Market regimes

Each day is labeled from trailing SPY data only. The first matching rule wins:

Regime	Rule
Bear / crash	SPY trailing 126-day return below −10%
High volatility	Not bear, and SPY trailing 21-day realized volatility above 25% annualized
Bull	Neither of the above, and SPY trailing 126-day return above +10%
Sideways	Everything else

When it raises a finding

Medium finding when the worst regime with at least 126 days has a Sharpe below −0.3 and a lower return than the benchmark over the same days.

Test 7

Look-ahead bias

on_bar() cannot see future prices by construction. prepare() can, so when a strategy has one it is tested:

Truncation test. prepare() is recomputed with all data after a cut date deleted, at cuts 30%, 50%, 70% and 90% of the way through the history. Its outputs before each cut are compared with the full-history outputs. Any change proves that past values depend on future data.
Point-in-time re-run. If look-ahead is confirmed, the strategy is re-run with prepare() recomputed on every bar from data up to that bar only. The report shows both results.

When it raises a finding

Critical finding when look-ahead is confirmed. The score is capped at 20 and every performance-based test is marked "Not valid".
Static look-ahead patterns (see Other checks) are reported as code notes. They are heuristics; the truncation test is the proof.

Test 8

Resampled histories

The strategy's daily net returns are resampled 1,000 times with a circular block bootstrap. The block length is the cube root of the number of days, between 5 and 63 days, so short-range patterns and volatility clustering are kept and overlapping days are never treated as independent. The random seed is fixed. Reports show 90% ranges (5th to 95th percentile) for the Sharpe ratio, CAGR and maximum drawdown. These ranges assume the future resembles this history; they are not forecasts.

When it raises a finding

Medium finding when the Sharpe ratio is positive but the lower end of its 90% range is at or below zero.

Test 9

Random timing

A strategy earns money two ways: by how much it is invested, and by when. This test checks the "when". The strategy's own positions are rotated against the market by 1,000 offsets of at least 63 trading days. Each rotation keeps the exposure, the holding periods and the trading costs exactly; only the alignment with what the market did next is broken. Good timing should beat most rotations. This is a Monte Carlo permutation test (Masters 2006; Aronson, Evidence-Based Technical Analysis, 2007). When you declare a development period, it uses the years after it, which your tuning never saw. It is not run when exposure never changes (buy & hold has no timing to test).

When it raises a finding

High finding when the strategy beats fewer than 5% of rotations: its timing is worse than random.
Medium finding below 50%: no better than random.
Low finding below 95%: some evidence, not conclusive. At 95% or more it is listed as a strength.
With no skill, the percentile is equally likely to land anywhere from 0 to 100%, so these thresholds are significance levels. In our calibration the coin-flip strategies landed between 3% and 90%.

Test 10

Market exposure

How much of the return is simply the market's? Daily net returns are regressed on SPY's to give beta, annualized alpha (Jensen 1968) and alpha's t-statistic, with Newey-West standard errors because overlapping positions make daily errors correlated. The findings about the edge over buy & hold of the same assets, which caps the score, are reported under this test too.

When it raises a finding

High finding when buy & hold of the same assets beat the strategy by more than 0.2 Sharpe; medium when it beat it at all; low when the strategy's edge is under 0.1.
Medium finding when alpha is −1% a year or worse with a t-statistic of −2 or below.
Low finding when beta is at least 0.3 and alpha's t-statistic is between −1 and 1: the returns are mostly market exposure.

Test 11

Walk-forward

The parameter sets that were already run (your values and each ±10% and ±20% change) are re-ranked every year by their Sharpe over the previous five years. The best one is traded for the next year, and the test years are stitched together. Every choice uses only data available at the time, so the stitched record is what re-tuning once a year would have produced. It re-tunes within that small neighborhood only, so it can't see a wider search you ran before uploading; the research-selection test covers that. It needs about eight years of trading history.

When it raises a finding

High finding when the full backtest's Sharpe is above 0.3 and the walk-forward Sharpe is at or below zero.
Medium finding when the walk-forward Sharpe is below half the full backtest's, or when fewer than half the test years have a positive Sharpe.

Test 12

Research selection

If you backtest many versions of a strategy and keep the best, its Sharpe is inflated by the selection itself. You say roughly how many variants you tried (rules, parameter values, assets, date ranges). The deflated Sharpe ratio (Bailey and López de Prado, 2014) then estimates the chance that the true Sharpe is above zero, allowing for that many tries, the length of the record, skew and fat tails. Only you know the number, so the test runs only when you give it (under "Your setup"; random shards choose the setup for you, so it doesn't apply there), and it does not change the score. Without it, the report says the selection risk is unknown.

When it raises a finding

High finding when the chance is below 50%.
Medium finding when it is below 95%.

Random shards (default mode)

A single backtest depends on the setup you chose: the assets, the dates and the cost. In random-shard mode, the default, you choose nothing, and the score comes from many random slices of history instead. The result is the Quantgrounds score.

A shardOne asset and one window. The asset is the whole US market or one of 49 industry portfolios from the Ken French Data Library, value-weighted over every NYSE/AMEX/NASDAQ stock including delisted ones, so there is no survivorship bias. The window is 2 to 8 years long, starts anywhere back to 1926, and shards overlap.
How many2,000 per run, drawn with a random seed that the report prints. The same seed reproduces the run exactly.
ScoringEach shard is scored with the same six sub-score formulas as a normal report, from slices of backtests that ran once over each asset's full history, at 5 basis points per side, against buy & hold of that asset over the same days. There is no development period, so risk-adjusted return and edge use the shard's Sharpe. Generalization compares the other assets over the same dates.
Quantgrounds scoreThe usual weights applied to each sub-score's average over all shards. Confirmed look-ahead caps it at 20.
Run-to-run spreadThe ± figure is 1.96 standard deviations of the headline across 300 resamples of the shards. In our tests, 20 reruns of each sample strategy stayed within a 2.5-point range (about ± 1.3). The report and the website show the same whole number and spread; halves round up.
StatusFail when fewer than 50% of shards have a positive Sharpe; Warning below 70%. At 80% or more it is listed as a strength.
DetailsThe rest of the report describes a fixed reference setup: the whole US market over its full history (1926 to today), no development period, 5 basis points per side. The report also prints that setup's own diagnostic score for reference; it is not the Quantgrounds score and usually differs from it.

Other checks

Code reviewStatic checks for negative shifts, centered rolling windows, backward fills and np.roll (look-ahead patterns); full-history statistics in prepare(); indexing later elements; parameters that are modified, undeclared or unused; unusually specific parameter values; many tunable parameters; hard-coded numbers and symbols; and silently swallowed errors. Each note has a line number.
DeterminismThe baseline is run twice. Different positions produce a high finding, because the results cannot be reproduced.
LeverageA low finding notes that requested weights were scaled down to a gross exposure of 1.
Trade countFewer than 30 trades: low finding. Fewer than 10: medium.
ConcentrationMedium finding when the five best trades produce more than 50% of net trade P&L, or the best trade more than 25%.
DrawdownsMaximum drawdown worse than −25%: medium finding; worse than −35%: high. Longest time below a previous peak over three years: medium.
BenchmarkA low finding when both the Sharpe ratio and the CAGR are below buy and hold of the same assets over the same period, with the same costs.

Statuses and verdict

Each row of the stress-test panel takes the most severe finding attached to that test:

Most severe finding	Status
Critical or high	Fail
Medium	Warning
Low	Note
None	Pass

A test that could not be applied (for example, no development period or no numeric parameters) is marked "Not run". When look-ahead is confirmed, every test except the look-ahead test and the code review is marked "Not valid".

The verdict is one sentence chosen by rule: it describes the test behind the most important critical or high finding. If there is none, it says no major weaknesses were found in these tests and how many tests have warnings.

Scores

There are two names for the same formula, so a report always says which one it is:

Quantgrounds scoreRandom-shard mode (the default). The formula below, applied to the average sub-scores over 2,000 random shards, with a ± spread. The PDF says "random shards" next to it.
Diagnostic score"Your setup" mode. The formula below for the one setup you picked. It tells you about that setup only, so it is not comparable with a Quantgrounds score.

Both are a weighted mean of six sub-scores, each from 0 to 100, then two caps. Values between the listed points are interpolated linearly. Every sub-score that looks at returns is judged against buy & hold of the same assets over the same days, because holding is free: a strategy that is half in cash, or flips a coin about being invested, earns a share of the market's return without any skill.

Edge over buy & hold (25%)Out-of-sample net Sharpe minus buy & hold's Sharpe over the same days (full period if there is no usable out-of-sample period of at least 252 days): −0.5 → 0, −0.25 → 15, 0 → 40, +0.25 → 65, +0.5 → 85, +1.0 or more → 100.
Risk-adjusted return (15%)The same net Sharpe on its own: 0 → 0, 0.5 → 50, 1.0 → 80, 1.5 or more → 100.
Parameter robustness (15%)100 × (half the median share of Sharpe kept across all changes, plus half the worst share kept at ±10%), with shares clipped to 0–1. Zero if the baseline Sharpe is not above 0.05.
Cost resilience (15%)Break-even cost as a multiple of your cost: 1x → 0, 2x → 40, 3x → 60, 5x → 80, 10x or more → 100. Zero if there is no return before costs.
Generalization (15%)100 × (half the share of held-out assets with a positive Sharpe, plus half the ratio of held-out to declared median Sharpe, clipped to 0–1).
Drawdown at equal risk (15%)The strategy is scaled to buy & hold's volatility and its worst drawdown is divided by buy & hold's: 0.3 → 100, 0.5 → 90, 0.7 → 70, 1.0 → 40, 1.5 or more → 0. Sitting in cash shrinks drawdowns and returns alike, so it earns nothing here; avoiding crashes does.

score = min( Σ weight × sub-score ÷ Σ weight used, 50 + 100 × edge )

Edge capThe score can't exceed 50 + 100 × the edge over buy & hold in Sharpe (never below 10). A strategy that doesn't beat holding tops out at 50; reaching 90 needs a Sharpe 0.4 above holding, on average over every random shard. In random-shard mode the edge is averaged over all shards.
Look-ahead capIf look-ahead is confirmed, the score is capped at 20.

Sub-scores that cannot be computed (for example, no numeric parameters) are left out and the remaining weights are re-normalized. Scores are shown as whole numbers, with halves rounded up, and the band comes from the number shown.

Score	Band	What it means
90–100	Exceptional	Rare. A large edge over buy & hold that survives every test. We have not seen a public rule reach it.
80–89	Strong	A clear edge over buy & hold that holds up across the tests.
65–79	Promising	Some edge that holds up in most tests.
45–64	Mixed results	Little or no edge over buy & hold, or an edge with notable weaknesses. Buy & hold itself scores about 50.
25–44	Weak	Does worse than buy & hold, or fails several tests.
0–24	Serious issues	Loses to buy & hold by a wide margin, or the backtest is invalid (look-ahead).

The score summarizes how the strategy did in these tests. It is not a measure of trading skill or future profitability.

How we check the score

A score is only useful if higher scores go with better results on data the score never saw. We check that with 19 strategies: published rules (trend filters, time-series momentum, volatility targeting, breakouts, short-term mean reversion), buy & hold, and deliberate failures (coin flips, an always-half-invested portfolio, a rule picked from 972 backtests, an overfit strategy and one with look-ahead). Each is scored in random-shard mode on the Ken French universe, cut at the end of 2004 and, separately, the end of 2010. We then measure what it actually did afterwards on a different universe — 15 to 18 real stocks and SPY — over 2005–2010 (including the financial crisis) and 2011–2018.

Higher scores tracked better results on the stocks the score never saw. The rank correlation of score with the following years' Sharpe was +0.61 for 2005–2010 and +0.57 for 2011–2018; a strategy the market beat by more than 0.2 Sharpe scored below the rest in 88% and 82% of pairs. The controls land where they should, and the floor is where the first, absolute-performance formula failed most: it once ranked future losers above the rest more often than not and handed its top score to the always-invested control. Now:

Strategy	Score 2004 / 2010	What it is
Fast moving-average crossover	59 / 59	trend rule with a small edge over holding the whole market
200-day trend filter	57 / 57	trend rule
Buy & hold	50 / 50	the benchmark itself
Always 50% invested	50 / 50	exposure without skill
Look-ahead bug	20 / 20	capped: it uses the future
Best of 972 mined backtests	10 / 10	data-mined (the old formula missed it, scoring ~47)
Daily coin flip	10 / 10	random

The strongest published rules — trend-following across the whole market — clear 50 by a few points because timing a diversified universe genuinely beat holding it through 2008; none reaches the upper bands, which require a large, robust edge. The move to the survivorship-bias-free French universe is what lets the score catch data-mining and mean-reversion rules that flattered themselves on a handful of survivor stocks. Future returns over six to eight years are noisy, the 2011–2018 correlation is lower than 2005–2010, and 19 strategies are a small sample, so these checks show the direction of the evidence, not proof. The strategies, the harness and the full results are in the calibration/ folder of the repository.

We ran the same check on the random-timing and market-exposure tests before deciding whether they belong in the score. Both predicted the following years' edge only weakly and inconsistently (rank correlations between +0.13 and +0.35), so they are reported as tests and left out of the score: they explain how a strategy makes money, not whether it beats holding.

Limitations

Daily closing prices only, so fills are modeled at the next close, with a flat cost and no market impact, borrow fees or taxes.
The random-shard universe (the default score) is survivorship-bias-free and runs to 2026, but the small stock set behind "Your setup" is a survivor set ending in 2018 (stocks) and 2019 (SPY).
Regimes use simple SPY rules for every asset.
The static code checks are heuristics and can miss problems or flag harmless code.
The random-timing percentile and the deflated Sharpe ratio are the only probability-like figures. The deflated Sharpe ratio is only as good as the number of variants you report, and rotations of one history are not independent samples. The other tests are not independent, so no combined p-value is given.
Walk-forward re-tunes within a small neighborhood of your parameters. A strategy found by a much wider search can pass it; the research-selection test is the guard for that.
Passing every test does not show that a strategy has an edge or will make money.

Sandbox

Strategy code that passes the pre-check runs in a separate process with restricted built-ins, an import allowlist, CPU, memory and time limits, file writes blocked and network access blocked. Each report's code review section records the limits that applied to that run and whether the process ran in an isolated network namespace. Code is passed to the process in memory, never written to disk, and discarded when the run ends. Results are kept in memory for up to an hour so the PDFs can be downloaded.

Verifying a report

A PDF is easy to edit. Each report therefore carries two things that tie its score to the exact code it tested:

Code fingerprintSHA-256 of the strategy file, after removing a leading byte-order mark, converting line endings to \n, stripping trailing spaces and tabs on each line, and dropping blank lines at the start and end. So the fingerprint survives copy and paste and editor differences, but any change to the code itself gives a different one. The report prints the first 16 characters on every page and the full value in the appendix.
Replay codeThe first 16 characters of the fingerprint, every setting that affects the score, and the score shown. Random shards: 1/fingerprint/S/seed/score. Your setup: 1/fingerprint/C/assets/development start/development end/cost/score, with - when there is no development period. Runs are deterministic: the same code, settings and engine version always give the same score.

To check a report, paste the code on the verify page. The fingerprint is computed in your browser. The page then re-runs the strategy with the replay code's settings, like any other run, and compares the score. A match shows the report's score is what this exact code gets; no key, account or stored record is involved, so any report can be checked, including ones made with the command-line tool. It can't show who wrote the code, whether it was copied, or how it will do in future. A report from an older engine version may not reproduce exactly if the engine or data has changed since.