Scope of the library

What belongs in spicefault and what is left to other tools. A feature request or a pull request can be checked against this page.

This is the scope of the software. The scope of the manuscript, which is narrower, is in Scientific scope and is not changed by it.

The criterion

Something belongs in the library if either holds:

  1. It needs the experiment engine and its provenance to be done right. Drawing or designing the samples, building each netlist, simulating in parallel and resumably, keeping the failed simulations, and recording what produced every row.
  2. It defines a quantity that every study would otherwise implement differently. A metric with a written definition, an estimator, an interval and a test against a population with a known answer.

What pandas, SciPy, SALib or scikit-learn do on the exported table stays outside, and is documented here as a recipe. The library has no dependency on any of them.

Inside

Area What
Fault injection Fault types as netlist primitives, the fault universe and its coverage
Healthy population Variations and tolerances, also correlated and by manufacturing lot
Experiments Drawn samples, designed samples and tolerance corners, operating conditions, the nominal circuit, a given parameter vector
Sampling Independent streams per sample; Latin hypercube and Sobol designs
Execution Parallel, resumable campaigns in which a failure is a row with a status
Datasets Provenance, verification, reproduction, specifications and pass/fail labels
Observation The instrument model: noise, resolution and range applied after the simulation
Statistics Statistics, quantiles, yield and margins of a fault-free population
Reliability Detectability, failure probability, diagnostic coverage, separability and ambiguity

Some of these are planned and not yet released: the roadmap tracks them.

Outside, with a recipe

Screening which parameters matter

The realised value of every varied parameter is a column of the dataset, so a rank correlation with a measurement is one line. It is a screening measure: it ranks monotonic effects and says nothing about interactions.

samples = dataset.samples[dataset.ok]
samples[list(dataset.parameters)].corrwith(samples["gain_db"], method="spearman")

Variance-based global sensitivity indices need their own sample matrix. The library simulates the points that SALib asks for, with Experiment.evaluate:

"""Recipe: variance-based sensitivity indices with SALib.

SALib chooses the points; the library simulates each one (`Experiment.evaluate`); SALib
computes the indices. The first-order index of a parameter is the share of the variance
of the measurement that it explains alone, and the total index includes its
interactions. Both are estimates with a confidence half-width, which SALib reports and
which shrinks slowly: n (2 d + 2) simulations for d parameters.

    pip install "spicefault[recipes]"
    python examples/recipes/sensitivity_indices.py
"""

import sys
import warnings
from pathlib import Path

import numpy as np

sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from rc_study import circuit, config, measurements  # noqa: E402
from SALib.analyze import sobol as analyze  # noqa: E402
from SALib.sample import sobol as sample  # noqa: E402

from spicefault import Experiment  # noqa: E402

TOLERANCE = {("R1", "value"): 0.01, ("C1", "value"): 0.05, ("R2", "value"): 0.01}
experiment = Experiment(circuit, config=config, measurements=measurements)
nominal = circuit.parameters()
targets = list(TOLERANCE)
problem = {
    "num_vars": len(targets),
    "names": [component for component, _ in targets],
    "bounds": [
        [nominal[t] * (1 - TOLERANCE[t]), nominal[t] * (1 + TOLERANCE[t])] for t in targets
    ],
}

if __name__ == "__main__":
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        points = sample.sample(problem, 64, calc_second_order=False, seed=0)
    gains = np.array([
        experiment.evaluate(dict(zip(targets, point, strict=True)))["nominal"]
        .measurements["gain_1k"]
        for point in points
    ])  # fmt: skip
    indices = analyze.analyze(problem, gains, calc_second_order=False, seed=0)
    print(f"{len(points)} simulations; indices of the gain at 1 kHz:")
    for name, first, total, half in zip(problem["names"], indices["S1"], indices["ST"],
                                        indices["ST_conf"], strict=True):  # fmt: skip
        print(f"  {name}: first order {first:5.2f}, total {total:5.2f} (+-{half:.2f})")
    print("the capacitor explains most of the variance:",
          bool(indices["ST"][1] > max(indices["ST"][0], indices["ST"][2])))  # fmt: skip

Searching for the worst case

Tolerance corners are inside the library. A search by optimisation is not: it is scipy.optimize around Experiment.evaluate, and its result is the worst case found, which may be a local extreme. Neither a search nor a finite random sample proves a worst case.

"""Recipe: search for the worst case of a measurement with SciPy.

The library simulates a given parameter vector (`Experiment.evaluate`); the search is
SciPy's. What it returns is the worst case *found*: a bounded quasi-Newton search stops
at a local extreme, and nothing here proves that there is no worse point. For a
response that is monotonic in each parameter over the tolerance band, as this one, the
worst case is a corner of the band, and the search is checked against it.

    pip install "spicefault[recipes]"
    python examples/recipes/worst_case.py
"""

import sys
from itertools import product
from pathlib import Path

import numpy as np
from scipy.optimize import minimize

sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from rc_study import circuit, config, measurements  # noqa: E402

from spicefault import Experiment  # noqa: E402

TOLERANCE = {("R1", "value"): 0.01, ("C1", "value"): 0.05, ("R2", "value"): 0.01}
experiment = Experiment(circuit, config=config, measurements=measurements)
nominal = circuit.parameters()
targets = list(TOLERANCE)


def gain(deviation) -> float:
    """The gain at 1 kHz with each parameter at nominal (1 + tolerance * deviation)."""
    values = {
        t: nominal[t] * (1 + TOLERANCE[t] * d) for t, d in zip(targets, deviation, strict=True)
    }
    return experiment.evaluate(values)["nominal"].measurements["gain_1k"]


if __name__ == "__main__":
    count = 0

    def objective(deviation):
        global count
        count += 1
        return gain(deviation)

    found = minimize(objective, x0=np.zeros(len(targets)), bounds=[(-1, 1)] * len(targets),
                     method="L-BFGS-B", options={"eps": 1e-2})  # fmt: skip
    corners = {corner: gain(corner) for corner in product((-1.0, 1.0), repeat=len(targets))}
    worst_corner = min(corners, key=corners.get)
    print(f"nominal gain at 1 kHz: {gain(np.zeros(len(targets))):.5f}")
    print(f"lowest gain found by the search: {found.fun:.5f} after {count} simulations")
    print("at deviations (in tolerances):", np.round(found.x, 3).tolist())
    print(f"lowest gain over the 8 corners: {corners[worst_corner]:.5f} at {list(worst_corner)}")
    reached = abs(found.fun - corners[worst_corner]) < 1e-6
    print("the search reached the worst corner:", bool(reached))

Both recipes are in examples/recipes/ and run with the tests; their packages come with pip install "spicefault[recipes]".

Confusion matrices and classifiers

Dataset.to_ml returns NumPy arrays and the splits keep the replicas of a circuit together, so the evaluation of a classifier is scikit-learn's:

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix

X, y = dataset.to_ml(target="fault_location")
train, test = split_by_replica(dataset.samples[dataset.ok], test_fraction=0.3, seed=0)
model = RandomForestClassifier(random_state=0).fit(X[train], y[train])
confusion_matrix(y[test], model.predict(X[test]))

The detection, escape and false-reject rates of a decision rule, with their intervals, are part of ReliabilityAnalysis.

Temperature coefficients

SPICE models them in the netlist, and an OperatingCondition sets the temperature:

R1 in out 10k tc1=100u tc2=0
OperatingCondition("hot", temperature=85)

Outside

  • Adaptive sampling and rare-event estimation. Research methods whose samples depend on earlier results, which conflicts with a plan that is fixed, resumable and the same for any number of workers.
  • Ageing and time-dependent reliability. They need failure models and field data that the library cannot validate. A drift of known size is already a parametric fault or a variation.
  • A graphical interface. It could be a separate application on the public interface.
  • Other simulators, until a study needs one: the backend is an interface.
  • Training of models. The library produces the dataset.