Structural brain alterations in chronic pain conditions remain incompletely understood, especially in chronic visceral pain. Patients with chronic-inflammatory or functional bowel disorders experience recurring abdominal pain in concert with other gastrointestinal symptoms, such as altered bowel habits, which are often exacerbated by stress. Despite growing interest in the gut-brain axis and its underlying neural mechanisms in health and disease, abnormal brain morphology and possible associations with visceral symptom severity and chronic stress remain unclear. We accomplished parallelized whole-brain voxel-based morphometry analyses in two patient cohorts with chronic visceral pain, i.e., ulcerative colitis in remission and irritable bowel syndrome, and healthy individuals. In addition to analyzing changes in gray matter volume (GMV) in each patient cohort vs. age-matched healthy controls using analysis of covariance (ANCOVA), multiple regression analyses were conducted to assess correlations between GMV and symptom severity and chronic stress, respectively. ANCOVA revealed reduced GMV in frontal cortex and anterior insula in ulcerative colitis compared to healthy controls, suggesting alterations in the central autonomic and salience networks, which could however not be confirmed in supplemental analyses which rigorously accounted for group differences in the distribution of sex. In irritable bowel syndrome, more widespread differences from healthy controls were observed, comprising both decreased and increased GMV within the sensorimotor, central executive and default mode networks. Associations between visceral symptoms and GMV within frontal regions were altered in both patient groups, supporting a role of the central executive network across visceral pain conditions. Correlations with chronic stress, on the other hand, were only found for irritable bowel syndrome, encompassing numerous brain regions and networks. Together, these findings complement and expand existing brain imaging evidence in chronic visceral pain, supporting partly distinct alterations in brain morphology in patients with chronic-inflammatory and functional bowel disorders despite considerable overlap in symptoms and comorbidities. First evidence pointing to correlations with chronic stress in irritable bowel syndrome inspires future translational studies to elucidate the mechanisms underlying the interconnections of stress, visceral pain and neural mechanisms of the gut-brain axis.