EMMmA

MESSENGER Environment

The MESSENGER environment is designed as a multi-task grid navigation game in which the agent need to retrive the message from a specific entity and deliver it to a target entity while avoiding the enermy entities

An instance of the environment contains task-relevant free-form textual manual where each item describes

1. the role of an entity (message, target or enermy)
2. the movement dynamics of an entity (stationary, chasing or fleeing)

Different from other similar environments, the mapping from entities to manual $\mathcal{M} : \mathcal{Z} \mapsto \mathcal{E}$ isn’t available

EMMmA Model

For a MESSENGER game $(\mathcal{S},\ \mathcal{A},\ \Omega,\ \mathcal{P},\ \mathcal{O},\ \mathcal{R} \mid \mathcal{E},\ \mathcal{Z},\ \mathcal{M})$, the policy network $\pi_{\theta}(a_{t} \mid o_{t - k : t},\ \mathcal{Z})$ consists of

Text Encoder

Each manual item $z \in \mathcal{Z}$ is encoded to a sequence of tokens $t_{1:n}$ through a pretrained BERT-based model. The key vector $k_{z}$ and value vector $v_{z}$ of manual item $z$ are obtained through the following transformation

$\begin{matrix} k_{z} = \sum_{i = 1}^{n} \alpha_{i} W_{k} t_{i} + b_{k} & \alpha = \operatorname{softmax}_{i}(u_{v} \cdot t_{i}) \\[7mm] v_{z} = \sum_{i = 1}^{n} \beta_{i} W_{v} t_{i} + b_{v} & \beta = \operatorname{softmax}_{i}(u_{k} \cdot t_{i}) \end{matrix}$

Entity Query

Each entity $e \in \mathcal{E}$ is assigned with an embedding $q_{e}$ as a query vector to obtain adaptive representation $x_{e}$

$x_{e} = \sum_{z \in \mathcal{Z}} \gamma_{z} v_{z} \quad \gamma = \operatorname{softmax}_{z} \left( \frac{q_{e} \cdot k_{z}}{\sqrt{d}} \right)$

The representation $x_{e}$ is further plugged into a tensor $X \in \mathbb{R}^{h \times w \times d}$ at the same position of $e$ on the grid map

Action Output

The categorical distribution of actions is calculated based on the queried representation tensors

$\pi_{\theta}(a_{t} \mid o_{t - k : t},\ \mathcal{Z}) = \operatorname{softmax}(\operatorname{FFN}(\operatorname{Flatten}(\operatorname{Conv2D}(X_{k}))))$

where the representation tensors of the $k$ recent observation are concatenated into $X_{k} \in \mathbb{R}^{h \times w \times kd}$