Project information
- Category: Deep Learning
- Type: Artificial Neural Networks for Classification tasks
- Client/Purpose: This experiment was aimed to learn and compare Deep Learning models using the MABe challenge data and tasks
- Project date: November, 2021
- Project URL: github
Classifying behaviors of multiple agents with Deep Learning models
The MABe Challenge is a competition hosted by AIcrowd, focused on evolving the understanding of such multi agent behavior prediction systems in order to remove the human component from behavior classifications. Specifically, in this challenge, a video of two mice has been encoded and mapped into a frame-by-frame coordinate system throughan automated software pipeline titled MARS. The resulting data set has been used to train neural networks to reliably predict behaviors. The experiments performed intended to create and submit models that could reach the baseline of two of the tasks in the MABe Challenge.
For this project, development of a supervised learning model for the task of Behavior Classification with the goal of learning feature representations was completed through several models for comparison: basic LSTM, LSTM-CNN, CNN, ResNet and Transformer. Then, with this pre-trained model, the goal was to perform new behavior learning on the task of predicting seven new unsupervised behaviors.
Several Semi-supervised approaches were implemented with combinations of the base models, fine-tuning Head backend models that were later used as a teacher to generate pseudo-labels on the test set, later supporting an interactive approach, described in the figure.
For the unsupervised learning task of new behaviors, two low dimensional embeddings methods were used: TSNE plots and Multi-Dimensional Scaling methods (MDS).
This would allow for the visualization of the clustering of arbitrary behaviors and give insight into the effectiveness of learned features.