On a question of Külshammer for homomorphisms of algebraic groups

Let G be a linear algebraic group over an algebraically closed field of characteristic p≥0. We show that if H1 and H2 are connected subgroups of G such that H1 and H2 have a common maximal unipotent subgroup and H1/Ru(H1) and H2/Ru(H2) are semisimple, then H1 and H2 are G-conjugate. Moreover, we show that if H is a semisimple linear algebraic group with maximal unipotent subgroup U then for any algebraic group homomorphism σ:U→G, there are only finitely many G-conjugacy classes of algebraic group homomorphisms ρ:H→G such that ρ|U is G-conjugate to σ. This answers an analogue for connected algebraic groups of a question of B. Külshammer.
In Külshammer's original question, H is replaced by a finite group and U by a Sylow p-subgroup of H; the answer is then known to be no in general. We obtain some results in the general case when H is non-connected and has positive dimension. Along the way, we prove existence and conjugacy results for maximal unipotent subgroups of non-connected linear algebraic groups. When G is reductive, we formulate Külshammer 's question and related conjugacy problems in terms of the nonabelian 1-cohomology of unipotent radicals of parabolic subgroups of G, and we give some applications of this cohomological approach. In particular, we analyse the case when G is a semisimple group of rank 2.