Group-convolutional architectures, which encode symmetries as inductive bias, have shown great success in applications, but can suffer from instabilities as their depth increases and often struggle to learn long range dependencies in data (e.g., over-smoothing in GNNs). We propose and study unitary group convolutions, which allow for deeper networks that are more stable during training. The main focus of the paper are graph neural networks, where we show that unitary graph convolutions provably avoid over-smoothing and achieve competitive performance on benchmark datasets compared to state-of-the-art graph neural networks. We complement our analysis with the study of general unitary convolutions and analyze their role in enhancing stability in general group convolutional architectures.