Housing Minister Sean Fraser will not be running in the next federal election, adding to a growing list of cabinet ministers exiting federal politics.