Advanced Digital Signal Processing for High Spectral Efficiency Multidimensional Transmission

Doctoral thesis, 2024

Modern coherent optical communication systems utilize all available physical dimensions for data modulation. In general, independent modulation and signal processing are performed in coherent systems, leaving a performance gap to the Shannon limit. This thesis focuses on advanced digital signal processing to achieve joint encoding and joint processing systems with high spectral efficiency (SE).

First, we investigate the low-complexity and practical digital predistortion approaches to eliminate the nonlinear transmitter distortion, which is one dominant impairment for the high-SE transponder. Joint processing of multiple wavelength channels in the receiver can eliminate the linear and nonlinear interchannel interference in long-haul superchannel systems. Using the frequency-locked received channels provided by the comb-based superchannel and receiver, two joint processing schemes are studied. We investigate the performance of multichannel equalization, which cancels linear interchannel crosstalk and reduces the guard bands, in the long-haul transmission constrained by the amplified spontaneous emission noise. In addition, we propose a perturbative-based compensation to eliminate both self-phase modulation and cross-phase modulation caused by the interference wavelength channels.

Multidimensional (MD) Vononoi constellations generated by a structured geometric shaping method provide shaping gain over conventional quadrature amplitude modulation (QAM) and have low-complexity encoding and decoding algorithms, which are suitable for high-SE applications. The use of MD formats closes the gap to the theoretical Shannon limit. By employing a 24-dimensional VC with a record constellation size of 7.9×10^28 and lookup table based predistortion, we present a 12.2 bit/s/Hz C-band transmission over 40 km single-mode fiber and have the first experimental demonstration of a significant MD shaping gain over QAM formats in the soft-decision coded system.