We have measured the optical response of detectors designed for SAFARI, the far-infrared imaging spectrometer for the SPICA satellite. To take advantage of SPICA's cooled optics, SAFARI's three bolometer arrays are populated with extremely sensitive (NEP∼2×10-19 W/√Hz) transition edge sensors with a transition temperature close to 100 mK. The extreme sensitivity and low saturation power (∼4 fW) of SAFARI's detectors present challenges to characterizing them. We have therefore built up an ultra-low background test facility with a cryogen-free high-capacity dilution refrigerator, paying careful attention to stray-light exclusion. Our use of a pulse-tube cooler to pre-cool the dilution refrigerator required that the SAFARI Detector System Test Facility provide a high degree electrical, magnetic, and mechanical isolation for the detectors. We have carefully characterized the performance of the test facility in terms of background power loading. The test facility has been designed to be flexible and easily reconfigurable with internal illuminators that allow us to characterize the optical response of the detectors. We describe the test facility and some of the steps we took to create an ultra-low background test environment. We have measured the optical response of two detectors designed for SAFARI's short-wave wavelength band in combination with a spherical backshort and conical feedhorn. We find an overall optical efficiency of 40% for both, compared with an ideal-case predicted optical efficiency of 66%.