EquineML: Machine learning methods for recognition of the pain expressions of horses

Recognition of pain in horses and other animals is important, because pain is a manifestation of disease and decreases animal welfare. Pain diagnostics for humans typically includes self-evaluation and location of the pain with the help of standardized forms, and labeling of the pain by an clinical expert using pain scales. However, animals cannot verbalize their pain as humans can, and the use of standardized pain scales is challenged by the fact that animals as horses and cattle, being prey animals, display subtle and less obvious pain behavior - it is simply beneficial for a prey animal to appear healthy, in order lower the interest from predators. The aim of this project is to develop a method for automatic recognition of pain in horses. The method employs an RGB-D sensor mounted in the stable ceiling, and detects and recognizes behavioral patterns related to pain, in an automated manner when the horse perceives itself as being alone. The Machine Learning-based pain classification system is trained with examples of behavioral traits labeled with pain level and pain characteristics. This automated, user independent system for recognition of pain behavior in horses will be the first of its kind in the world. A successful system might change the concept for how we monitor and care for our animals.


Robotics, Perception and Learning
KTH Royal Institute of Technology, Sweden
Project Director: Hedvig Kjellstr÷m
Clinical Sciences
Swedish University of Agricultural Sciences, Sweden
Principal Investigator: Pia Haubro Andersen


Via forskning kan datorer se hästens 3D-form, Hästsverige, January, 2021.


Zhenghong Li, Sofia Broomé, Pia Haubro Andersen, and Hedvig Kjellström. Automated detection of equine facial action units, arXiv:2102.08983, 2021.
Computer vision

Joonatan Mänttäri*, Sofia Broomé*, John Folkesson, and Hedvig Kjellström. Interpreting video features: A comparison of 3D convolutional networks and convolutional LSTM networks. In Asian Conference on Computer Vision, 2020. (*Joint first authors)
Computer vision

F. M. Serra Bragança, S. Broomé, M. Rhodin, S. Björnsdóttir, V. Gunnarsson, J. P. Voskamp, E. Persson-Sjodin, W. Back, G. Lindgren, M. Novoa-Bravo, C. Roepstorff, B. J. van der Zwaag, P. R. Van Weeren, and E. Hernlund. Improving gait classification in horses by using inertial measurement unit (IMU) generated data and machine learning. Nature Scientific Reports 10, paper number 17785, 2020.
Gait analysis

Katrina Ask, Marie Rhodin, Lena-Mari Tamminen, Elin Hernlund, and Pia Haubro Andersen. Identification of body behaviors and facial expressions associated with induced orthopedic pain in four equine pain scales. Animals 10(11), 2155, 2020.
Pain diagnostics

Maheen Rashid, Hedvig Kjellström, and Yong Jae Lee. Action graphs: Weakly-supervised action localization with graph convolution networks. In IEEE Winter Conference on Applications of Computer Vision, 2020.
Computer vision

Sofia Broomé, Karina Bech Gleerup, Pia Haubro Andersen, and Hedvig Kjellström. Dynamics are important for the recognition of equine pain in video. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Computer vision

Pia Haubro Andersen, Karina B. Gleerup, Jennifer Wathan, Britt Coles, Hedvig Kjellström, Sofia Broomé, Yong Jae Lee, Maheen Rashid, Claudia Sonder, Erika Rosenberg, and Deborah Forster. Can a machine learn to see horse pain? An interdisciplinary approach towards automated decoding of facial expressions of pain in the horse. In International Conference on Methods and Techniques in Behavioral Research, 2018.
Pain diagnostics