In this paper we derive a novel fourth-order gauge-invariant phenomenological model of infinitesimal rate-independent gradient plasticity with isotropic hardening and Kröner’s incompatibility tensor inc ( ε p ) : = Curl [ ( Curl ε p ) T ] , where ε p is the symmetric plastic strain tensor. Here, gauge-invariance denotes invariance under diffeomorphic reparametrizations of the reference configuration, suitably adapted to the geometrically linear setting. The model features a defect energy contribution that is quadratic in the tensor inc ( ε p ) and it contains isotropic hardening based on the rate of the plastic strain tensor ε · p . We motivate the new model by introducing a novel rotational invariance requirement in gradient plasticity, which we call micro-randomness, suitable for the description of polycrystalline aggregates on a mesoscopic scale and not coinciding with classical isotropy requirements. This new condition effectively reduces the increments of the non-symmetric plastic distortion p · to their symmetric counterpart ε · p = sym p · . In the polycrystalline case, this condition is a statement about insensitivity to arbitrary superposed grain rotations. We formulate a mathematical existence result for a suitably regularized non-gauge-invariant model. The regularized model is rather invariant under reparametrizations of the reference configuration including infinitesimal conformal mappings.