We visited Laugardalur Park after visiting the zoo (which is not very good) but this place more than made up for it. There are a number of different pools of varying sizes and depths. When we visited it was -9 but sitting in the smaller pools was still very warm and relaxing. As it was so cold some of the bigger pools were a bit cold to lounge around in but you could still do lengths. There is also a very good water slide although getting out of the pool was very cold.
This place is much cheaper than the blue lagoon but does not have the same wow factor and if you have time/money you should visit both. This place is nice to chill after being busy all day but it is a half hour walk out of town, although I doubt a taxi would cost much to get here.