Autoencoders for Physical-Layer Communications: Approaches and Applications

Doctoral thesis, 2023

The ever-growing demand for higher data rates has driven continuous developments in communication systems over the years. As upcoming high-bandwidth services require even higher data rates, future digital communication infrastructures must undergo continuous upgrades to provide increased capacity. Recently, machine learning has surfaced as a potential tool to augment this capacity further. A particularly promising avenue lies in the application of autoencoders. These can concurrently optimize both the transmitter and receiver tailored to a specific channel model and performance metric, a paradigm commonly referred to as end-to-end autoencoder learning.

In this thesis, we study different aspects of using machine learning for physical-layer communications, spanning wireless and optical communication in terms of applications, and unsupervised, supervised, and reinforcement learning in terms of methodologies. The main contributions of this thesis are listed as follows.

Firstly, to overcome the challenge that standard end-to-end autoencoder learning requires a differentiable channel model for gradient-based transmitter optimization, Paper A and Paper B explore reinforcement learning-based transmitter optimization. In Paper A, considering that reinforcement learning-based training necessitates sending a feedback signal from the receiver to the transmitter, we propose a novel method for the  feedback signal quantization. Simulation results demonstrate that the proposed quantization scheme facilitates effective transmitter learning with limited feedback. In Paper B, reinforcement learning is applied to mitigate transmitter hardware impairments. A novel digital predistorter based on neural networks is introduced and trained in a back-to-back optical fiber transmission experiment. Experimental results demonstrate that the proposed digital predistorter effectively mitigates transmitter impairments, outperforming commonly used baseline schemes.

Secondly, Paper C and Paper D focus on supervised learning, with an emphasis on improving the interpretability of end-to-end autoencoder learning-based communication systems. In Paper C, a novel model-based autoencoder is proposed for nonlinear systems. By decomposing the autoencoder-based transceivers into concatenations of smaller neural networks, the proposed method allows for the visualization of each learned functional block, improving the interpretability of the learned transmission scheme. Paper D interprets the learned solution from a different perspective by carefully selecting baseline schemes. We demonstrate that, for the linear systems considered in Paper D, machine learning methods do not significantly outperform conventional model-based approaches. Instead, they learn invertible transformations of these model-based solutions.

Lastly, Paper E focuses on unsupervised learning, addressing the problem of blind channel equalization for both linear and non-linear channels. By introducing a constraint to the latent representation of a standard autoencoder, a novel autoencoder-based blind equalizer is formulated. Simulation results demonstrate that, for both linear and nonlinear channels, the proposed equalizer can achieve similar performance as conventional data-aided equalizers while outperforming state-of-the-art blind methods.