Outputs: the result object and tables
Source:vignettes/scphasedcn-outputs.Rmd
scphasedcn-outputs.Rmdcall_scn() returns a scphasedcn object;
write_scphasedcn(res, out) writes it as flat tables (the
command line does both). read_scphasedcn(out) reads a
written run back.
Files
| file | content |
|---|---|
cells.tsv |
one row per called cell: cell type, depth stratum, null cell flag,
n_calls, n_events, and the
multiplet flag (normal mode always; tumour mode when
multiplets are given: the cells above its
multiplet_prob); tumour mode adds clone,
posterior, doublet_like,
ref_pool
|
calls.tsv |
per cell x region calls (normal mode: the called rows; tumour mode: every clone row and every per-cell inherited / confirmed row) |
pooled_calls.tsv |
normal mode: called rows of the cell pools (named event
regions only with pool_events, off by default:
experimental, not calibrated per event) |
events.tsv |
a cell’s overlapping called regions merged into events (normal mode:
with null_cell, p, q,
null_borrowed; q is descriptive, not a false
discovery rate, see False-event
load) |
thresholds.tsv |
the null thresholds per score, region class and depth stratum (and
cross-validation fold); named event regions have no null
rows of their own and use the thresholds of their sd_class
(column borrowed_from, present when a run has events; the
pooled ones only with pool_events) |
regions.tsv |
the regions scored (arms, chromosomes, 5-50 Mb tiles, events; tumour mode also clone pieces) |
null_fit.tsv |
the depth null (Student-t per class x stratum) and the allelic
overdispersion rho
|
false_calls.tsv, false_events.tsv
|
normal mode: the realised false-call rate on the null cells |
expected_false.tsv |
normal mode: expected false calls and events per class x state
(event_fdr(); lower bounds) |
null_selection_check.tsv |
normal mode with call_scn(null_check = TRUE) /
--null-check: the selection-free check of the per-cell null
(null_selection_check()) |
clones.tsv, assignments.tsv,
clone_segments.tsv, clone_profiles.tsv.gz,
reference_pool.txt
|
tumour mode: clones, per-cell assignment, clone copy-number segments, clone window profiles, the normal reference cells |
summary.tsv |
headline numbers (scphasedcn_summary()) |
params.yaml |
every parameter of the run; reads back with
caller_params("params.yaml")
|
NOTES.md |
the run notes (inputs, steps, run time, package version) |
result.rds |
the object without the large scores /
features tables |
scores.tsv.gz, features.tsv.gz
|
optional (write_scores, write_features;
large) |
Calls
| column | meaning |
|---|---|
cell |
the cell (normal mode pools: the pool id; tumour mode clone rows: the clone) |
region, class, chrom,
start, end
|
the region (0-based, half-open) and its class: arm,
chrom, 5Mb … 50Mb,
event
|
state |
loss (1:0), gain (2:1), cnloh
(2:0, copy-neutral LOH), gain_as, amp,
del, neutral, or a clone state label |
score, thr, called
|
the region score, its null threshold and
score > thr
|
level |
cell, pool or clone
|
source |
cell (the cell’s own evidence), inherited
(from its clone or pool) or confirmed (a clone call the
cell’s own evidence confirms) |
reportable |
the call survives the nesting rules (e.g. a chromosome gain is not reported inside a stronger nested arm loss) |
resolved |
normal mode, per cell: why resolve_nested() changed the
row (NA = unchanged): no_depth (a chromosome / event gain
without depth support), nested <region> <state>
(a gain explained by a stronger nested loss / CN-LOH),
replaces <region> gain (the nested call made
reportable in its place) |
gate_z, gate_thr
|
normal mode with gain_gate: "z": on the gated
chromosome / event gains, the depth z and its threshold (the larger of
the midpoint between neutral and the gain shift and the
gain_gate_q quantile of the depth null); a gain has depth
support when gate_z > gate_thr. With the default
gain_gate: "dl" the gate is the sign of the gain depth
log-likelihood ratio (z above the midpoint) and these columns are
absent |
z, f, n_hap
|
normal mode, cell rows: depth z, major-haplotype fraction and molecule votes of the cell in the region (not in tumour mode) |
stratum |
normal mode, cell rows: the cell’s depth stratum (clean fragments;
<20k, 20-50k, >=50k; not in
tumour mode) |
p, q, null_borrowed
|
normal mode, tested per-cell rows: the empirical p-value against the null rows, the row-level Benjamini-Hochberg q-value (descriptive: not a false discovery rate, not a filter) and whether the row’s chromosome (chrX) or class has no null rows of its own (see below); NA on other rows |
confident |
tumour mode: the cell’s clone assignment is confident |
False-event load (normal mode)
Normal mode calls a cell-region when its score exceeds the 1 / 1,000
quantile of the null rows of its class and depth stratum
(null_q). The budget is per cell-region, and a cell has
about a hundred reportable rows (chromosomes, arms, 50 Mb tiles), so
among thousands of cells a run carries a predictable number of false
per-cell events. event_fdr() (run by
call_scn()) makes that number visible; it changes no call,
threshold or score.
Expected false calls and events
(expected_false.tsv, one row per class x state x chromosome
group):
| column | meaning |
|---|---|
class, state
|
the reportable per-cell class x state (e.g. chrom
gain, arm loss) |
chrom_group |
A (autosomes), X, Y
|
null_borrowed |
the rate comes from the autosomal null rows (chrX rows are never
null rows), or from the sd_class of a class without null
rows |
n_rows |
per-cell rows of the class (every cell x region; each one a null hypothesis) |
n_tested |
of these, the rows whose best state is this state and that would be reportable if called (calling policy, gain depth gate) |
n_called, n_events
|
reportable called rows of this state; events whose top region is of this class and state |
n_null, n_null_called,
n_null_events, null_per_1000
|
null rows of the class, how many of them are reportable calls of this state (fold-cross-validated thresholds, policy and gain gate applied as for every cell), how many null-cell events have such a top region, and the call rate per 1,000 null rows |
expected_false_calls,
expected_false_events
|
the null rate per depth stratum x n_rows of that
stratum, summed over the strata; _lo / _hi: 95
% Poisson interval of the null count |
false_frac_calls, false_frac_events
|
expected false / called (capped at 1): the implied false proportion (a lower bound, see below) |
These are lower bounds. Every row counts as a null
hypothesis, but the null rate is measured on the selected null cells,
which were chosen for having no chromosome or arm call at a stricter
threshold (select_null_cells()). Their rows are cleaner
than those of an unselected normal cell; when the parametric selection
null is too thin (shared allelic bias, noisier cells), the selection
truncates the null tail and the realised false rate of the other cells
is higher. chrX rows (and their events) take the autosomal rate, which
ignores X-inactivation driven allelic imbalance in females.
summary.tsv carries the headline:
events | cell_events_in_null_cells (events found in the
null cells themselves; their chrX is not screened),
events | expected_false_cell_events (and per chromosome
group), events | expected_false_cell_event_fraction,
calls | expected_false_reportable_cell_calls.
p- and q-values. For every tested per-cell row
(classes reportable per cell, plus 20 Mb rows promoted by the nesting
rule) p compares the score of a reportable call
(the score when the row would be reportable if called, else minus
infinity) with the same statistic on the null rows of the row’s
threshold group, in the cross-validation folds its threshold was trained
on: p = (1 + #{null >= score}) / (n + 1) (a conformal
p-value). A named event region borrows the thresholds of
its sd_class, and its rows are compared with the same null
rows (those of the sd_class group and folds), scored as event rows would
be (null_borrowed = TRUE). With n null rows
p cannot go below 1 / (n + 1); a strong event
would then get a q-value that only reflects how many other events are as
strong (with m tests no single event can reach a q below
about m / (n + 1), which is above 1 on real data). Beyond
the (51st) largest null score of the group, p therefore
follows an exponential tail fitted to the top 50 null scores times the
reportable share of that tail (event_fdr(tail = "none")
gives the plain conformal p-value; summary.tsv
events | empirical_p_floor reports the floor).
q is the Benjamini-Hochberg q-value over all tested rows,
called or not. In events.tsv, p /
q / null_borrowed are those of the event’s top
region and null_cell says whether the event is in a
selected null cell. q is a descriptive column: it
is not a false discovery rate and must not be used as a filter.
Two independent limits make it too small, and no event-level q-value is
computed:
-
Relative to the selected null.
p, and the exponential tail beyond the largest null scores, are computed against the selected null cells. The selection removes the cells with the heaviest tails (e.g. cells with more allelic overdispersion than modelled), so for cells like thosepis too small and the tail model extrapolates a truncated distribution; the tail is conservative only for a correctly specified null with light tails. In a sample where part of the cells are noisier than the null cells, the default tail can give false events a smallqthatevent_fdr(tail = "none")does not (on a simulated null in which 10 % of the cells had a higher allelic overdispersion, 28 events reachedq< 0.05, all false; none withtail = "none"); compare the two when the null is in doubt. -
Rows, not events. Benjamini-Hochberg runs over rows, and
one real event usually rejects several nested rows (chromosome, arms,
tiles), while rows of one cell are positively dependent. A cut on
qinevents.tsvis therefore not an event-level false discovery rate: the share of false events below the cut can be well above it (in a simulation with 300 planted chromosome losses in 3,000 cells, 12 % of the events withq< 0.05 were false). The same holds forsummary.tsvevents | cell_events_q_below_0.05, a descriptive count of events at a row-levelq.
Selection-free check
(null_selection_check.tsv,
call_scn(null_check = TRUE) / --null-check; it
costs about one more scoring pass). The autosomes are split into odd and
even chromosomes; null cells are selected on one half only, thresholds
are trained as usual on that half’s null rows, and exceedances are
counted on the other half of the same cells, rows that neither the
selection nor the thresholds saw; then the halves swap. Per class x
stratum (and all): n_rows,
n_exceed, per_1000 with a 95 % Clopper-Pearson
interval (lo_per_1000, hi_per_1000; rows of
one cell overlap, so the interval is somewhat narrow),
nominal_per_1000 (1000 x (1 - null_q)),
insample_per_1000 (the cross-validated rate on the
selection half of the same passes) and cv_per_1000 (the
run’s own in-sample rate, false_calls.tsv). Cells are
selected on one half only, so a cell with a real event on the measured
half stays in and that event’s rows count as exceedances:
per_1000 is the null rate plus the rate of real events on
the measured half, an upper bound of the null rate. A rate clearly above
the nominal and in-sample rates has two possible causes: a selected null
that is cleaner than the cells it is applied to (the expected false
counts above are then too low), or real events. Read it next to the
share of cells with a called event (cells.tsv
n_events, summary.tsv): in a sample without
real events the excess measures the selection; in a sample with them it
is an upper bound. The halves carry different chromosomes, so a class
whose regions differ in their null spread shows up here as well.
Clone segments (tumour mode)
clone_segments.tsv: per clone and segment
chrom, start, end,
n_win (1 Mb windows), the pooled oriented votes
nA, nB, the total copy number cn,
the allele-specific state (1:1, 1:0,
2:1, 2:0, 3:1, …),
log2 (depth relative to the clone baseline), f
(major-haplotype fraction), llr, confident,
label (neutral, loss, cnloh, gain, gain_as, amp, del) and
ploidy (1 on a male’s chrX / chrY). The clone
N0 is the fixed normal clone.
assignments.tsv: per cell the likelihood-best clone, its
posterior, the margin to the second clone, whether the cell
looks like a doublet of two clones (doublet_like,
pair, p_mix), and confident
(posterior >= 0.9, not doublet-like).
The object
library(scphasedcn)
library(data.table)
sim = simulate_scn(n_cells = 60, sex = "female", cell_types = c(a = 0.5, b = 0.5), events = NULL,
multiplet_rate = 0, depth = c(meanlog = log(12000), sdlog = 0.3), phase_block_mean = 1e9,
switch_rate = 2e-6, seed = 3)
d = file.path(tempdir(), "out_extract")
invisible(write_sim_dataset(sim, d))
cen = sim$scn$bins[, .(chrom, cstart = 1, len = max(end)), by = chrom][, .(chrom, cstart, cen_left = 0.4 * len,
cen_right = 0.4 * len, cend = len)]
res = call_scn(d, "normal", sex = "female", centromeres = cen, null_cells = "select", verbose = FALSE)
names(res)
#> [1] "mode" "cells" "regions" "features"
#> [5] "scores" "thresholds" "calls" "events"
#> [9] "pools" "clones" "assignments" "segments"
#> [13] "profiles" "null_fit" "params" "notes"
#> [17] "false_calls" "false_events" "expected_false" "fdr_info"
#> [21] "embedding" "feature_info" "run"
res$thresholds[level == "cell" & fold == 0][1:5, .(score, class, stratum, q, thr, n_null)]
#> score class stratum q thr n_null
#> <char> <char> <char> <num> <num> <int>
#> 1: s_depth 10Mb <20k 0.999 4.0226146 780
#> 2: s_allelic 10Mb <20k 0.999 3.1379661 780
#> 3: s_comb 10Mb <20k 0.999 -0.8174322 780
#> 4: s_depth 20Mb <20k 0.999 3.6923150 480
#> 5: s_allelic 20Mb <20k 0.999 2.8782689 480
out = file.path(tempdir(), "out_caller")
invisible(write_scphasedcn(res, out))
res2 = read_scphasedcn(out)
identical(res2$calls, res$calls)
#> [1] FALSEThe components of the object:
| component | normal mode | tumour mode |
|---|---|---|
cells |
cell, cell_type, stratum, fold, null_cell, multiplet | the cells with clone, posterior, … |
regions |
the scored regions | the scored regions plus clone pieces |
scores |
per cell x region scores (with the null flag) | NULL |
thresholds |
per-cell (cross-validated) and pool thresholds
(level) |
per depth stratum and clone hypothesis |
calls, events
|
see above | see above |
pools |
pool membership (kind = type or
knn) |
NULL |
clones, assignments,
segments, profiles
|
NULL | see above |
null_fit |
depth t parameters, rho, the null cells and how they were selected | depth t parameters, rho, reference pool, clone discovery details |
false_calls, false_events,
expected_false, fdr_info,
null_check
|
null false-call rates, expected false calls / events, the q-value
settings (tests, tail, calibration groups), the selection check (with
null_check = TRUE) |
NULL |
params |
all parameters, including the normal: caller
arguments |
all parameters, including the tumour: options |
features, feature_info,
embedding, notes
|
the per-cell region features and the run notes | the same |
Allelic tile mask (normal mode, opt-in)
With mask_allelic = TRUE (overdispersion rule) and / or
mask_f0 = TRUE (shared-bias rule) in the parameters,
call_scn() selects null cells once, assesses every
autosomal 5 Mb tile on their votes (allelic_noisy_tiles()),
and leaves the votes of the masked tiles out of the allele features
before calling (depth is untouched). The two rules can be used alone or
together (the union is masked); the run notes give the number of tiles
each rule masked. null_fit$allelic_mask holds the per-tile
table:
| column | meaning |
|---|---|
tile, chrom, start,
end
|
the 5 Mb tile |
n_rows, votes
|
null rows (cells) of the tile and their molecule votes |
rho, z_rho
|
method-of-moments beta-binomial overdispersion of the null votes, and its robust z of log rho over the assessed tiles |
assessed |
the tile has at least mask_min_votes votes per null
row |
masked |
the tile’s votes were left out (the active rules) |
f0, d0, z_f0
|
vote-weighted haplotype fraction of the null rows
sum(nA) / sum(n_hap), its deviation
abs(f0 - 0.5), and the robust z (median / MAD) of
d0 over the assessed tiles |
masked_rho |
overdispersion rule: assessed and
z_rho > mask_rho_mad
|
masked_f0 |
bias rule: assessed, z_f0 > mask_f0_mad and
d0 > mask_f0_min
|
masked_rho and masked_f0 are reported
whether or not their rule is on; the run notes list the masked tiles.
The bias rule masks tiles with a shared allelic imbalance among the null
cells, such as a stretch off 1:1 in every cell (germline copy-number
variant or segmental duplication, imprinted or allele-specific
accessible region, mapping bias, false heterozygous SNPs). A real clonal
event that is frequent enough for its carriers to remain among the null
cells looks the same and can be masked: in a simulation with a focal
event in 35 % of cells, 60 of 96 carriers stayed in the null set and all
four event tiles were masked. The mask is therefore a diagnostic to be
read with the list of masked tiles, not a filter to apply blindly to
samples that may carry clonal events.
allelic_bias_tiles(features) computes the same table over
all cells of a features object, without a caller run, to look at shared
bias before calling. Under mirrored orientation
(phasing = "mirrored") the haplotype fraction is the
per-block major fraction, which measures coverage rather than shared
bias: call_scn() then runs without the bias rule and says
so in the run notes.