Private Next Location Prediction using Transformers: Enhancing Accuracy under Differential Privacy Constraints

Next-location prediction is an essential task in applications where personalizing users' experience is a key factor. In recent years, deep learning techniques trained on large datasets are the method of choice for prediction tasks. In particular, large language models (LLMs) such as transformers are proving to be very effective for this prediction task. However, training on users' personal data, in particular user trajectories, raises serious privacy threats, as models may memorize mobility patterns and increase the risk of attacks that attempt to reconstruct individual information and movement behaviors. We study the problem of training transformers on individual location data under differential privacy guarantees, and we identify the causes why existing techniques achieve insufficient accuracy. We propose a series of techniques that enhance the overall accuracy of the trained model, while maintaining the same (epsilon, delta) privacy constraints. Our experimental evaluation shows that our approach achieves a higher accuracy than benchmarks, under the same privacy constraints, particularly under stricter privacy regimes (epsilon <= 1). In fact, the accuracy of our approach is similar to that of models trained without privacy protection.