SELF-PRUNING NEURAL NETWORK
BRIEF
Neural networks can contain many parameters. This experiment investigates whether learned gating can increase sparsity while retaining useful predictive performance.
ENGINEERING IDEA★
Build a custom gate-controlled neural network in PyTorch using custom PrunableLinear layers and evaluate different sparsity penalties across controlled training runs.
TOOLS USED
Python · PyTorch · Neural Networks · CIFAR-10 · Sparsity Regularization
PROJECT TYPE
CIFAR-10 Input Vectorization
32×32×3 RGB image tensors flattened and normalized for feed-forward classification.
PrunableLinear Gated Weights
Each weight matrix W is paired with learnable gate parameters g, producing effective weights via sigmoid gating σ(g).
Joint Cross-Entropy + L1 Gate Penalty
Loss function combines classification loss with λ · ||σ(g)||₁ to drive non-essential connection gates toward zero.
Threshold Pruning & Evaluation
At λ = 1e-4, documented test accuracy reaches 54.19% while achieving 33.42% weight sparsity.
- ★Designed custom PyTorch nn.Module subclass (PrunableLinear) maintaining separate weight and gate parameter tensors.
- ★Applied sigmoid activation to gate scores during the forward pass to compute effective weights dynamically.
- ★Formulated an L1 regularization term over active gate probabilities scaled by hyperparameter lambda (λ).
- ★Evaluated sparsity-versus-accuracy trade-offs across multiple regularization settings on the CIFAR-10 benchmark.
- ✓Documented experimental result at λ = 1e-4: 54.19% test accuracy on CIFAR-10 with 33.42% connection sparsity.
- ✓Demonstrated how learnable gating integrates directly into gradient-based optimization without manual layer-by-layer heuristics.
- ✓Established clear empirical trade-off curves showing how stronger L1 penalties increase sparsity while impacting classification capacity.
- →Balancing task loss against sparsity regularization requires careful schedule tuning so gates do not collapse before features stabilize.
- →Future extensions could apply PrunableConv2d layers to convolutional backbones for higher baseline vision accuracy on CIFAR-10.