On 17th February 2021, Eduardo Sanchez defended his thesis by zoom. Awarded by Toulouse III – Paul Sabatier University, his work was realized both at IRIT and IRT Saint Exupéry, under the supervision of EDMITT doctoral school.
The high quality and the relevance of Eduardo’s thesis contributed to get significant results as a part of Synapse project.
THESIS SUBJECT
“Learning disentangled representations of satellite image time series in a weakly supervised manner.”
About this thesis
This work focuses on learning data representations of satellite image time series via an unsupervised learning approach. The main goal is to enforce the data representation to capture the relevant information from the time series to perform other applications of satellite imagery.
However, extracting information from satellite data involves many challenges since models need to deal with massive amounts of images provided by Earth observation satellites. Additionally, it is impossible for human operators to label such amount of images manually for each individual task (e.g. classification, segmentation, change detection, etc.). Therefore, we cannot use the supervised learning framework which achieves state-of-the-art results in many tasks.
To address this problem, unsupervised learning algorithms have been proposed to learn the data structure instead of performing a specific task. Unsupervised learning is a powerful approach since no labels are required during training and the knowledge acquired can be transferred to other tasks enabling faster learning with few labels.
In this work, we investigate the problem of learning disentangled representations of satellite image time series where a shared representation captures the spatial information across the images of the time series and an exclusive representation captures the temporal information which is specific to each image. We present the benefits of disentangling the spatio-temporal information of time series, e.g. the spatial information is useful to perform time-invariant image classification or segmentation while the knowledge about the temporal information is useful for change detection.
To accomplish this, we analyze some of the most prevalent unsupervised learning models such as the variational autoencoder (VAE) and the generative adversarial networks (GANs) as well as the extensions of these models to perform representation disentanglement. Encouraged by the successful results achieved by generative and reconstructive models, we propose a novel framework to learn spatio-temporal representations of satellite data. We prove that the learned disentangled representations can be used to perform several computer vision tasks such as classification, segmentation, information retrieval and change detection outperforming other state-of-the-art models. Nevertheless, our experiments suggest that generative and reconstructive models present some drawbacks related to the dimensionality of the data representation, architecture complexity and the lack of disentanglement guarantees.
In order to overcome these limitations, we explore a recent method based on mutual information estimation and maximization for representation learning without relying on image reconstruction or image generation. We propose a new model that extends the mutual information maximization principle to disentangle the representation domain into two parts. In addition to the experiments performed on satellite data, we show that our model is able to deal with different kinds of datasets outperforming the state-of-the-art methods based on GANs and VAEs. Furthermore, we show that our mutual information based model is less computationally demanding yet more effective.
Finally, we show that our model is useful to create a data representation that only captures the class information between two images belonging to the same category. Disentangling the class or category of an image from other factors of variation provides a powerful tool to compute the similarity between pixels and perform image segmentation in a weakly-supervised manner.
SYNAPSE PROJECT
SYNAPSE project helps operators and integrators of space, to remain competitive with the new entrants of digital world. In a context of massive and open data production ; enabling the provision of a service to more users, at higher-frequency and more responsively ; enabling tools and methods mostly from the Big Data world for the emergence of new applications.
JURY
| Elisa FROMONT | Jury President | IRISA |
| Nicolas COURTY | Rapporter | IRISA |
| Ronan FABLET | Rapporter | IMT Atlantique |
| Mélanie DUCOFFE | Examiner | Airbus/IRT Saint Exupéry |
| Jean-Michel LOUBES | Examiner | IMT/Aniti |
| Mathieu SERRURIER | Thesis Director | IRIT/IRT Saint Exupéry |
| Mathias OTNER | Thesis Co-Director | Airbus Defence & Space |
PUBLICATIONS
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations of satellite image time series. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Würzburg, 2019.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Image-to-image translation for satellite image time series representation learning. In Conférence sur l’Apprentissage automatique (CAp), Toulouse, 2019.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Proceedings of the European Conference on Computer Vision (ECCV), Online, 2020.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Mutual information measure for image segmentation using few labels. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Online, 2020.