MACHINE LEARNING FOR MUSIC AND MOVEMENT COORDINATION - Reddy BN, Krishna; Kataria, Ansh; Shrivastava, Vijayendra Kumar; Grover, Madhur; Prasad, Durga; Kulkarni, Nishant (ShodhKosh: Journal of Visual and Performing Arts, 2025)
The coordination of music and movement is a complicated interaction of auditory perception with motor planning and instant sensorimotor combination. New developments in the field of machine learning have opened up new possibilities to model, predict and improve this interaction to be used in the area of performance analysis, rehabilitation, interactive systems and human-computer collaboration. In this paper, the researcher examines a multimodal model that combines audio characteristics alongside kinematic movement information to elicit temporal and spatial dynamics of coordinated behavior. Based on the prior experience in rhythm perception, beat tracking, and gesture recognition, the proposed system uses the latest deep learning models, such as CNNs, RNNs, LSTMs, and Transformers, to train effective representations of rhythmic shape and movement patterns. An end to end signal-processing chain is used, which has audio preprocessing, motion-capture or IMU-based trackers and filtering to minimize noise and guarantee reliability of the data. The process of feature extraction is temporal, spectral and kinematic, which allows the models to deduce the accuracy of synchronization, the quality of movement, and sensitivity to musical cues. The strategies of training focus on cross-validation, hyperparameter optimization and regularization to enhance generalization of various datasets and styles of movements. The findings indicate that the multimodal learning is more effective in predicting the beat alignment, the classification of gestures, and the time coordination as compared to the unimodal learning methods.