How do I stop the toffee on banoffee pie from going too hard?

I made a banoffee pie the cheats way (boiling the condensed milk for 5 mins - with sugar and er). Left it in fridge to set but final product was very hard and chewy !!