Spatiotemporal vision-language transformer for hand pointing direction estimation with zero-shot diffusion segmentation

Hand pointing is a fundamental nonverbal gesture used to establish joint attention and convey spatial intent. Despite its importance, existing recognition approaches often remain limited to binary detection or rely on specialized hardware, constraining their applicability in natural environments. This paper introduces a vision-language spatiotemporal transformer framework that integrates visual, textual, and temporal cues to estimate hand point direction. The approach leverages zero-shot diffusion segmentation to automatically generate upper-body masks without manual annotation, which are fused with input frames to enhance spatial focus on pointing regions. Spatial and temporal features are then encoded using transformer-based architectures to learn sequence-level visual features. Language-level learning is enabled by combining semantic textual embeddings from a vision-language model with visual features via a cross-attention mechanism. This synergistic design allows the model to jointly predict both pointing occurrence and direction vectors with high accuracy. Extensive experiments on benchmark datasets show that the proposed model consistently outperforms stateof-the-art baselines, achieving precision and recall above 90% and reducing mean angular error to less than 10 degrees. Moreover, the framework enables semantic prompt-guided inference on previously unseen video samples using text-encoded prompts, making it highly valuable for practical applications where labeled data or semantic context is unavailable.