The housing secretary says the government's plans to change wait times for permanent residency are unfair because they "change the goalposts".