Congratulations to the CIAR project for two space premieres:
– Remotely updating, from a ground station, an artificial neural network embedded on a satellite.
– Running this neural network on a FPGA (Field-Programmable Gate Array) in orbit.
The CIAR project (Chaîne Image Autonome et Réactive / / Responsive and Autonomous Imaging Chain) is studying technologies to deploy Artificial Intelligence (AI) for image processing on embedded systems (satellites, delivery drones, etc.). Three interdependent challenges are addressed:
- Defining use cases and building relevant image database.
- Designing AI that is both performant and well suited to embedded systems constraints.
- Optimizing the selected algorithm and embedding them in hardware targets.
For several years, the CIAR team has been collaborating with the European Space Agency (ESA) on the OPS-SAT mission. On March 22nd 2021, the team successfully uploaded a neural network on the FPGA embedded in this satellite. This illustrates the ability of IRT Saint Exupéry to tackle all these technical challenges.
Challenge #1: Defining the use case
OPS-SAT is a 3U (30x10x10 cm) nano-satellite launched on December 18, 2019. This ESA technology demonstrator aims to validate new on-orbit control systems. OPS-SAT embeds an Intel Cyclone-V computer (a “System On Chip” or SoC), as well as a small camera with a sampling distance of 50 meters.
Not all AI use cases are conceivable at this resolution, but one possible and useful service identified by the CIAR team is cloud detection in images. Indeed, pixels covered by clouds are unusable pixels for, let’s say, drawing maps or monitoring the evolution of the terrain. By identifying cloudy pixels with a good level of confidence, we can decide to remove them at the source in order to preserve the satellite’s memory and bandwidth.
From a practical standpoint, another advantage of cloud detection is that there are public databases that can be used to train AI algorithms for this use case. Once the satellite was in flight, the CIAR project was able to study how neural networks trained on reference data performed on real OPS-SAT images.
The trainings were refined with operational images to improve and update the algorithms remotely from the ground.
Challenge #2: The design of efficient AI
To obtain the best possible execution times, the project has chosen to use the programmable logic of the Cyclone-V (i.e. the FPGA) as the main computing resource.
The CIAR team had to design algorithmic solutions that were both accurate and compatible with the number of logic elements available on this very small satellite (110 kLE). Several types of architectures were developed:
- Classical neural networks (CNN) derived from Lenet-5 [3].
- Fully-convolutional neural networks (FCNs) [1, 3].
- Spiking Neural Networks (SNN) [4].
- Hybrid neural networks (partially formal, partially spiking) [2, 3].
- Genetic algorithms (ZGP, for “Zoetrope Genetic Programming”) [5].
The performance of these different architectures has been optimized, validated and compared before their deployment on hardware targets.
Challenge #3: Hardware optimization and implementation
If libraries exist to optimize and execute neural networks on CPU or GPU (Central / Graphics Processing Unit), porting these algorithms to FPGA targets is more complicated.
The network structure must be converted to RTL code and integrated into a custom environment. The CIAR project adopted a “pipelined and parallelized” approach to minimize execution time. The parameters (weight/bias) were also quantized from 32-bit floats to a fixed arithmetic compatible with the execution target.
In the end, the hardware optimization carried out by the CIAR project on the OPS-SAT satellite allows the on-orbit processing of 25 million pixels per second (inference time on the FPGA) with a power consumption lower than 2 watts.
neural architectures (from [3]).
Challenges ahead
With these firsts in orbit, IRT Saint Exupéry demonstrates how FPGAs reprogramming capabilities can improve Artificial Intelligence remotely and shift satellites to new missions after their launch
Also, the use of an FPGA allows combining precision of the algorithm, execution speed and low power consumption.
On board processing of images to extract useful information in real time will enable optimization, early warning and automatic reconfiguration functions from space. As long as OPS-SAT remains functional, the CIAR team plans to develop other AI algorithms and compare their performance in orbit. Further scientific publications are planned.
The project is also looking at other applications of AI to image processing, such as improving the detection and tracking of pedestrians moving around autonomous delivery drones.
Our partners
IRT Saint Exupéry would like to thank its academic and industrial partners involved in the CIAR project who made these successes possible:
Activeon is a software company offering innovative solutions for task scheduling, workload automation, scalability and operationalization of machine learning. For the CIAR project, Activeeon provided its expertise and off-the-shelf product “Machine Learning Open Studio (MLOS)” to help optimize neural architectures.
AViSTO is a software engineering company. It is specialized in complex projects using object-oriented, web and mobile technologies. To this core competency, AViSTO has progressively added peripheral skills to software development, including DevOps, quality assurance, data science and cybersecurity.
ELSYS Design is an engineering company specializing in the design of embedded systems. It supports innovative projects with high technical added value in the fields of embedded software, microelectronics, electronic boards and embedded systems. The sectors covered are space, aeronautics and defense.
Geo4i is a French company specialized in geospatial data management, geospatial information enhancement and imagery analysis in the fields of defense, security and economic intelligence. Geo4i’s expertise covers various fields such as imagery intelligence, geospatial intelligence, geospatial data management, data mining and open data.
Inria is a public scientific and technological institution. Its mission is to develop research and commercialization in information and communication sciences and techniques, both nationally and internationally. The institute also leads the French national strategy in terms of research in artificial intelligence.
The Laboratory of Electronics, Antennas and Telecommunications is a joint unit of the Nice Sophia Antipolis University and the CNRS. The LEAT develops its research around three themes: Modeling and System Design of Communicating Objects; Design and Modeling of Antennas; Microwave Imaging and Antenna Systems.
MyDataModels has developed the TADA platform, powered by evolutionary algorithms. The company is interested in Small Data, and allows professionals to obtain powerful predictive models from small amounts of data. MyDataModels wants to help professionals in their decision making and thus boost their performance.
Combining 40 years of experience and a unique diversity of expertise, talents and cultures, Thales Alenia Space architects, design and deliver high-tech solutions for telecommunications, navigation, Earth observation, environmental management, exploration, science and orbital infrastructure. Thales Alenia Space is a joint venture between Thales (67%) and Leonardo (33%).
TwinswHeel designs and manufactures autonomous delivery droids. The company’s droids provide concrete solutions to logistics problems at large industrial sites. TwinswHeel is also studying a solution based on an autonomous vehicle capable of delivering goods in urban areas throughout the year.
Références
[1] “Low-power neural networks for semantic segmentation of satellite images”, IEEE International Conference on Computer Vision (ICCV 2019). Gaetan Bahl, Lionel Daniel, Matthieu Moretti, and Florent Lafarge.
[2] “An FPGA-based Hybrid Neural Network accelerator for embedded satellite image classification”, IEEE International Symposium on Circuit and Systems (ISCAS 2020). Edgar Lemaire, Matthieu Moretti, Lionel Daniel, Benoit Miramond, Philippe Millet, Frederic Feresin, and Sébastien Bilavarn.
[3] “Onboard Image Processing using AI to reduce data transmission: example of OPS-SAT satellite”, 7th International Workshop on On-Board Payload Data Compression (OBPDC 2020 – Best Paper). Frédéric Férésin, Michael Benguigui, Yves Bobichon, Edgar Lemaire, Matthieu Moretti, and Gaétan Bahl.
[4] “Hardware design of spiking neural networks for energy efficient brain-inspired computing”, Phd thesis (Université Côte d’Azur 2020). Nassim Abderrahmane.
[5] “ZGP : une alternative aux réseaux de neurones pour la segmentation sémantique de nuages dans les images satellites multi-spectrales”, Conférence Nationale sur les Applications Pratiques de l’Intelligence Artificielle (APIA 2021). Ingrid Grenet, Yves Bobichon, Adrien Girard, and Frédéric Férésin.