We present our work on building and standardizing common abstractions and interchange formats (realized via MLIR) for Fully Homomorphic Encryption (FHE), a cryptographic technique that allows computation over encrypted data and has recently started to see real-world deployment (e.g., in Microsoft Edge). Developing FHE application poses significant challenges, and compilers have emerged to address these. Recently, the major players in the community have come together to unify the currently splintered ecosystem through the development of MLIR-based abstractions and interchange formats. We report on our abstractions (parts of which we plan to upstream) and our experiences of using MLIR as a vehicle for ecosystem unification and standardization, and suggest ways to improve.