Perceptual surface roughness classification describes how a surface's texture feels haptically in terms of perceptual categories such as smooth, rough, bumpy, etc. Computer vision and pattern recognition algorithms which estimate a surface's perceptual roughness have a wide range of application areas including robotics, assistive devices, telesurgery and teleperception. In this paper, we propose a novel approach to perceptual surface roughness classification that, unlike previous approaches, is designed to handle multiple roughness categories within the same image. The steps of our approach include (1) texton extraction and classification using a multi-class, non-linear Support Vector Machine; (2) segmentation using the Iterated Conditional Modes algorithm; and (3) overall perceptual roughness classification using a Nearest Neighbor classifier. The proposed approach is evaluated using visio-haptic subjective measures of roughness on images of the 3D texture of real world objects.