ga★
Garvit AroraPORTFOLIO 2026 — AI & SOFTWARE

SELF-PRUNING NEURAL NETWORK

Deep Learning / AI Research — Learnable Gating & Sparsity Regularization on CIFAR-10

BRIEF

Neural networks can contain many parameters. This experiment investigates whether learned gating can increase sparsity while retaining useful predictive performance.

ENGINEERING IDEA★

Build a custom gate-controlled neural network in PyTorch using custom PrunableLinear layers and evaluate different sparsity penalties across controlled training runs.

TOOLS USED

Python · PyTorch · Neural Networks · CIFAR-10 · Sparsity Regularization

PROJECT TYPE

*Deep Learning / AI Research
*Documented Research Experiment
01 // SYSTEM ARCHITECTURE & VISUAL SCHEMATIC
ARCH // PRUNABLE-LINEAR FEED-FORWARD (CIFAR-10)W_eff = W ⊙ σ(g) | L1 PENALTY λ = 1e-4
INPUT xGATE σ(g₁)GATE σ(g₂)LOGITS ŷDOCUMENTED REPORTCONFIG: λ = 1e-4WEIGHT SPARSITY33.42%TEST ACCURACY (CIFAR-10)54.19%--- PRUNED GATE (σ(g) → 0)─── ACTIVE WEIGHT CONNECTION
Regularization Strengthλ = 1e-4Documented report benchmark configuration
Test Accuracy54.19%Evaluated on CIFAR-10 test split at λ = 1e-4
Achieved Sparsity33.42%Pruned weight connections documented in report
02 // 4-STAGE WORKFLOW PROGRESSION
STAGE 01

CIFAR-10 Input Vectorization

32×32×3 RGB image tensors flattened and normalized for feed-forward classification.

STAGE 02

PrunableLinear Gated Weights

Each weight matrix W is paired with learnable gate parameters g, producing effective weights via sigmoid gating σ(g).

STAGE 03

Joint Cross-Entropy + L1 Gate Penalty

Loss function combines classification loss with λ · ||σ(g)||₁ to drive non-essential connection gates toward zero.

STAGE 04

Threshold Pruning & Evaluation

At λ = 1e-4, documented test accuracy reaches 54.19% while achieving 33.42% weight sparsity.

03 // IMPLEMENTATION, OUTCOMES & KEY LEARNINGS
IMPLEMENTATION DETAILS
  • ★Designed custom PyTorch nn.Module subclass (PrunableLinear) maintaining separate weight and gate parameter tensors.
  • ★Applied sigmoid activation to gate scores during the forward pass to compute effective weights dynamically.
  • ★Formulated an L1 regularization term over active gate probabilities scaled by hyperparameter lambda (λ).
  • ★Evaluated sparsity-versus-accuracy trade-offs across multiple regularization settings on the CIFAR-10 benchmark.
VERIFIED OUTCOMES
  • ✓Documented experimental result at λ = 1e-4: 54.19% test accuracy on CIFAR-10 with 33.42% connection sparsity.
  • ✓Demonstrated how learnable gating integrates directly into gradient-based optimization without manual layer-by-layer heuristics.
  • ✓Established clear empirical trade-off curves showing how stronger L1 penalties increase sparsity while impacting classification capacity.
REFLECTIONS & NEXT STEPS
  • →Balancing task loss against sparsity regularization requires careful schedule tuning so gates do not collapse before features stabilize.
  • →Future extensions could apply PrunableConv2d layers to convolutional backbones for higher baseline vision accuracy on CIFAR-10.