Radon transform on symmetric matrix domains

Artikel i vetenskaplig tidskrift, 2009

Let K = R, C, H be the field of real, complex or quaternionic numbers and Mp,q(K) the vector space of all p × q-matrices. Let X be the matrix unit ball in Mn-r,r(K) consisting of contractive matrices. As a symmetric space, X = G/K = O(n - r, r)/O(n - r) × O(r), U(n - r, r)/U(n - r) × U(r) and respectively Sp(n - r,r)/Sp(n - r) × Sp(r). The matrix unit ball y0 in Mr'-r,r with r' ≤ n - 1 is a totally geodesic submanifold of X and let Y be the set of all G-translations of the submanifold y0. The set Y is then a manifold and an affine symmetric space. We consider the Radon transform Rf(y) for functions f ∈ C ∞ 0(X) defined by integration of f over the subset y, and the dual transform Rt F(x),x ∈ X for functions F(y) on Y. For 2r < n, 2r ≤ r' with a certain evenness condition in the case K = R, we find a G-invariant differential operator M and prove it is the right inverse of RtR, RtR Mf = cf, for f ∈ C∞ 0(X), c ≠ 0. The operator f → RtR f is an integration of f against a (singular) function determined by the root systems of X and y0. We study the analytic continuation of the powers of the function and we find a Bernstein- Sato type formula generalizing earlier work of the author in the set up of the Berezin transform. When X is a rank one domain of hyperbolic balls in Kn-1 and y0 is the hyperbolic ball in Kr'-1, 1 < r' < n we obtain an inversion formula for the Radon transform, namely MRtR f = cf. This generalizes earlier results of Helgason for non-compact rank one symmetric spaces for the case r' = n - 1.