The ballistic sorting effect has been proposed to be a driver behind the observed size sorting on the rubble pile asteroid Itokawa. This effect depends on the inelasticity of slow collisions with granular materials. The inelasticity of a collision with a granular material, in turn, depends on grain size. Here we argue that determining the inelasticity of such collisions in an asteroid-like environment is a nontrivial task. We show non-monotonic dependency of the coefficient of restitution (COR) on target particle size using experiments in microgravity. Employing numerical simulations, we explain these results with the growing influence of adhesion for smaller-sized particles. We conclude that there exists an optimum impactor to target particle size ratio for ballistic sorting.