Justice Minister Sean Fraser rejected calls from five premiers for Ottawa to withdraw its legal argument calling for limits on the use of the Constitution's notwithstanding clause.