This study uses a global climate model (GCM) to investigate the climate response at the surface and in the atmosphere caused by land use change. The climate simulations are performed with the National Center for Atmospheric Research Community Land Model 3.5 (CLM3.5) coupled to the Community Atmosphere Model 3 (CAM3) and a slab ocean model. We use the Moderate Resolution Imaging Spectroradiometer (MODIS) surface albedo product to represent surface albedo in the CLM3.5 for both present day and to reconstruct the surface albedo for natural pre-agriculture conditions. We compare simulations including vegetation changes and surface albedo changes to simulations including only surface albedo changes. We find that the surface albedo change is most dominant in temperate regions while the change in evapotranspiration drives the climate response in the tropics. Our results show that land cover changes contribute to an annual global warming of 0.04 K, but there are large regional differences. In North America and Europe, the surface temperatures decrease by − 0.11 and − 0.09 K, respectively, while in India the surface temperatures increase by 0.09 K. When we fix the vegetation cover in the simulations and let the climate changes be driven only by the differences in surface albedo, the annual global mean surface warming is reduced, and all three regions are now associated with surface cooling. We also show that the surface albedo value for cropland is of major importance in climate simulations of land cover change. The surface albedo effect is the main driving mechanism when the change in surface albedo between agricultural and natural vegetation is substantial. Finally, we argue that differences in the surface albedo value of cropland implemented in earlier land use change studies explain the diversity in the sign and magnitude of the climate response.