Fluid-fluid displacement in porous and fractured media is an important process. Understanding and controlling this process is key to many practical applications, such as hydrocarbon recovery, geological storage of CO2, groundwater remediation, etc. Here, we numerically study fluid-fluid displacement in rough-walled fractures. We focus on the combined effect of wettability and fracture surface topography on displacement patterns and interface growth. We develop a novel numerical model to simulate dynamic fluid invasion under the influence of capillary and viscous forces. The capillary force is calculated using the two principal curvatures (aperture-induced curvature and in-plane curvature) at the fluid-fluid interface, and the viscous force is taken into account by solving the fluid pressure distribution. The aperture field of a fracture is represented by a spatially correlated random field, which is described by a power spectrum for the fracture wall topography and a cutoff wave-length.We numerically produce displacement patterns ranging from stable displacement, capillary fingering, and viscous fingering, as well as the transitions between them. We show that both reducing the aperture variability and increasing the contact angle (from drainage to weak imbibition) stabilize the displacement due to the influence of the in-plane curvature, an effect analogous to that of the cooperative pore filling in porous media. Implications of these results will be discussed.