With billions of devices connected over the internet, the rise of sensor-based electronic devices have led to cloud computing being used as a commodity technology service. These sensor-based devices are often small and limited by power, storage, or compute capabilities, and hence, they achieve these capabilities via cloud services. However, this gives rise to data privacy issues as sensitive data is stored and computed over the cloud, which at most times, is a shared resource. Homomorphic encryption can be used along with cloud services to perform computations on encrypted data, guaranteeing data privacy. While about a decade's work on improving homomorphic encryption has ensured its practicality, it is still several magnitudes slower than expected, making it expensive and infeasible to use. In this work, we propose a first-of-its-kind FPGA-based arithmetic hardware library that focuses on accelerating the key arithmetic operations involved in Ring Learning with Error (RLWE) based homomorphic encryption. We design and implement the FPGAbased Residue Number System (RNS), Chinese Remainder Theorem (CRT), modulo inverse and modulo reduction operations as a first step. For all of these operations, we include a hardware cost efficient serial, and a fast parallel implementation in the library. A modular and parameterized design approach helps in easy customization, provides flexibility to extend these operations for use in most homomorphic encryption applications, and fits well into emerging FPGA-equipped cloud architectures.