The impacts of climate change on the food system are a key concern for societies and policy makers globally. Assessments of the biophysical impacts of crop productivity show modest but uncertain impacts. But crop growth is not the only factor that matters for the food production. Climate impacts on the labour force through increased heat stress also need to be considered. Here, we provide projections for the integrated climate-induced impacts on crop yields and worker productivity on the agro-economy in a global multi-sector economic model. Biophysical impacts are derived from a multi-model ensemble, which is based on a combination of climate and crop models, and the economic analysis is conducted for different socio-economic pathways. This framework allows for a comprehensive assessment of biophysical and socio-economic risks, and outlines rapid risk increases for high-warming scenarios. Considering heat effects on labour productivity, regional production costs could increase by up to 10 percentage points or more in vulnerable tropical regions such as South and South-East Asia, and Africa. Heat stress effects on labour might offset potential benefits through productivity gains due to the carbon dioxide fertilisation effect. Agricultural adaptation through increased mechanisation might allow to alleviate some of the negative heat stress effects under optimistic scenarios of socio-economic development. Our results highlight the vulnerability of the food system to climate change impacts through multiple impact channels. Overall, we find a consistently negative impact of future climate change on crop production when accounting for worker productivity next to crop yields.