Given the widespread goal of endowing robotic systems with interactional capabilities that would allow users to deal with them intuitively by using means of natural communication, the text addresses the question to which extent it would be possible to mathematize (aspects of) social interaction. Using the example of a robotic museum guide in a real-world scenario, central challenges in dealing with the situatedness and contingency of human communicational conduct are shown using fine-grained video analysis combining the robot’s internal perspective with the user’s view. On a conceptual level, the text argues to consider human and robot as one ‘interactional system’ that jointly solves a practical (communicational) task. This opens up the perspective to integrate the human’s interactional competences and adaptability in the design and modeling of interactional building blocks for HRI. If we provide the technical system with systematic resources to make use of the human’s competences, the limits of mathematization might gain an interesting twist. Through careful design of the robot’s conduct, a powerful resource exists for the robot to pro-actively influence the users’ expectations about relevant subsequent actions, so that the robot could contribute to establishing the conditions which would be most beneficial to its own functioning.