An Adaptive Technique for Neural Network Training with Private Features and Public Labels

Differentially-private stochastic gradient descent (DP-SGD) represents the de-facto standard for privacy-preserving training of neural networks (NNs) under the differential privacy (DP) model. Its canonical formulation assumes that both the input features and the corresponding labels of training instances require protection. Newer developments explore scenarios in which only the labels are private, while the features are public. Doing so decreases the amount of required noise, leading to improved model accuracy. We investigate a complementary and underexplored setting where labels are non-sensitive, but the input features contain private information. Instead of perturbing gradients, our proposed methodology for training private NNs adds noise at a designated sanitization layer within the network. We analyze key architectural and algorithmic trade-offs inherent in this design and demonstrate how modifying the network architecture to reflect these considerations can lead to improved predictive performance. We also devise two adaptive algorithm optimizations: the first one identifies early stopping conditions in the learning process in order to save privacy budget and boost the protection strength; the second customizes the clipping threshold at each learning iteration in order to improve accuracy. Extensive experiments on real data show that our approach significantly outperforms the DP-SGD baseline.