Research

Motivation
Ancient turtles

 

Since I can remember I am a curious guy. My parents had this “I Wonder Why” type of book and I recall flipping through the book and being amazed at the answers to questions like: why we have days and nights? why dinosaurs no longer exist? why can birds fly? why can’t penguins fly? Over time, many questions emerged to which the book had no answer, and others for which I still can not get an answer to. However, the sense of curiosity has always followed me to this day, and it is a major driving force of my work. More specifically, one of the things that fascinates me most is “life”, which by definition, is “the condition that distinguishes organisms from inorganic objects and dead organisms, being manifested by growth through metabolism, reproduction, and the power of adaptation to environment through changes originating internally“.

 

Read full article »

Reinforcement Learning, Acting Under Uncertainty and Partial Observability
Futuristic Robot

 

My Ph.D thesis focused in developing flexible and robust mechanisms for autonomous agents by using the framework of (computational) reinforcement learning (RL). Within the field of machine learning, RL is the discipline concerned with providing mechanisms that allow an agent to accomplish a task through trial-and-error interactions with a dynamic and sometimes uncertain and unreliable environment. Having into account my research interests, what I find attractive within this field is its close relatedness with biological learning. However, despite being inspired by such mechanisms, the RL community has been more dedicated in designing algorithms that can efficiently learn “correct” courses of action given consecutive interactions with the environment, independently of the biological validity of the methods used.

Read full article »

The Reward Mechanism and Intrinsic Motivation
The Discovery of Brazil by Pedro Ávares Cabral

 

To tackle with some of the aforementioned challenges, in my Ph.D I focused on the reward mechanism embedded in the agent. In RL, the reward mechanism that the designer builds into the agent guides it throughout learning and implicitly defines the task that it must accomplish. It critically impacts both the time taken to learn a task and what is learned. As such, building a “good” reward mechanism is crucial for the performance of the agent. A major challenge when designing RL agents is then to build reward mechanisms that allow them to learn the intended task as efficiently as possible. Another challenge has to do with creating rewards that are generic enough to be used in a wide variety of situations, i.e., not only for a specific task or domain.

Read full article »

The Role of Emotions, Affective Computing and Emotion-based Reward Design
Animal Emotions

 

A great portion of my Ph.D thesis was dedicated to incorporating ideas from the emotional processing of humans and other animals into the framework of IMRL. Emotions are one of the most common behavioral phenomena observed in nature, yet they have often been considered as detrimental to rational and sound decision-making. However, as research in psychology, biology, ethology, neuroscience and other areas has shown, emotions are a beneficial adaptive mechanism for problem solving, enhancing perception, memory, attention and other cognitive skills. The need for an attention-focusing, interrupting mechanism having the properties of emotions has been advocated for a long time within AI by some of its founders (e.g., by Marvin Minsky or Herbert Simon).

Read full article »

Cooperation, Relatedness, Reciprocation and Socially-Aware Learning Agents
Cooperative Hunting

 

Another aspect of biological life that I find interesting is the fact that humans and other animals do not inhabit their environments by themselves: they live within social groups as a way to augment their survival chances, e.g., by hunting or performing complex actions together. Nevertheless, they still compete for resources or dispute social status as a way to improve their reproductive power. Throughout evolution, nature has also shaped social mechanisms that facilitate the way animals communicate their intentions and evaluate the actions of others, even in inherently competitive settings.

Read full article »

Classical Conditioning, Associative Trees and Learning in Factored MDPs
One of Pavlov's Dogs

 

Associative learning is a paradigm from the field of behaviorism that posits that learning occurs whenever a change in behavior is observed. Classical conditioning is one of the best-known associative learning paradigms and is one of the most basic survival tools found in nature by allowing organisms to expand the range of contexts where some of their already-known behaviors can be applied. By associating co-occurring stimuli from the environment, the organism can activate innate phylogenetic responses (e.g., fight or flight responses) to new and previously unknown situations.

Read full article »