WASHINGTON (AP) — The Supreme Court wrestled Wednesday with a case that could make it easier for the president to fire the head of the agency that oversees government-controlled mortgage giants Fannie Mae and Freddie Mac.

But the justices seemed skeptical about doing what shareholders of the companies have asked: setting aside an agreement under which the companies have paid the government more than $245 billion. That money was compensation for the taxpayer bailout Fannie and Freddie received after the 2007 housing market crash.

Justice Neil Gorsuch called the shareholders' ask "a big one" and "hard for us to swallow."

The case before the justices involves the Federal Housing Finance Agency, which oversees Fannie and Freddie and was created following the housing market crash. One of the questions for the court, which has been hearing arguments by phone because of the coronavirus pandemic, is whether the agency's structure violates the Constitution.

The case is in many ways similar to  one the justices decided earlier this year involving the FHFA's companion agency, the Consumer Financial Protection Bureau. The CFPB, the government's consumer watchdog agency, was created by Congress in response to the same financial crisis.

In the case involving the CFPB, the court struck down restrictions Congress imposed that said the president could only fire the agency's director for "inefficiency, neglect of duty, or malfeasance in office." Just as the head of the CFPB was, the head of the FHFA is nominated by the president and confirmed by the Senate to a five-year term. The director is then only removable by the president "for cause."

That structure can leave a new president with a director chosen by the previous president for some or all of the new president's time in office.

Several justices, however, seemed to think there was at least one wrinkle that makes this case different from the one it decided earlier this year. That's because the agreement Fannie and Freddie shareholders are complaining about was entered into when the FHFA was headed by an acting director who could have been removed by the president for any reason.

The case before the justices could have significant consequences.

In 2008, in response to the housing crisis, the FHFA put both Fannie and Freddie in so-called government conservatorships and arranged a government bailout that ultimately amounted to $187 billion. Four years later, the government and Fannie and Freddie adopted new terms of their financial agreement for paying the money back. Under that agreement, Fannie and Freddie gave the government nearly all of their profits each quarter as a dividend. The shareholders have objected to that agreement. They argue it should be set aside because it was imposed by an unconstitutional agency.

But Justice Elena Kagan and Justice Samuel Alito suggested that setting aside a past action in the current case could have vast consequences for another similarly structured agency, the Social Security Administration, and perhaps other agencies. Kagan suggested some 17 million past decisions of the Social Security Administration could be void.

"So are we really going to void all of those decisions?" she asked.

The Trump administration, for its part, is arguing that the shareholders are barred from challenging the agreement. But the government is not defending the constitutionality of the FHFA's structure, and the Supreme Court has appointed a lawyer to argue that position.

The two consolidated cases the court is hearing are Collins v. Mnuchin, 19-422, and Mnuchin v. Collins, 19-563.