WASHINGTON (AP) — House Speaker John Boehner's decision to take plum committee assignments away from four conservative Republican lawmakers after they bucked party leaders on key votes isn't going over well with conservative advocacy groups that viewed them as role models.

Reps. Tim Huelskamp of Kansas and Justin Amash of Michigan will lose their seats on the House Budget Committee chaired by Rep. Paul Ryan next year. And Reps. Walter Jones of North Carolina and David Schweikert of Arizona are losing their seats on the House Financial Services Committee.

The move is underscoring a divide in the Republican Party between tea party-supported conservatives and the House GOP leadership.

"This is a clear attempt on the part of Republican leadership to punish those in Washington who vote the way they promised their constituents they would — on principle — instead of mindlessly rubber-stamping trillion dollar deficits and the bankrupting of America," said Matt Kibbe, president of the tea party group FreedomWorks.

Michael Steel, a spokesman for Boehner, would only say Tuesday that the party's steering committee chaired by the speaker made the decision "based on a range of factors."

Groups aligned with the tea party movement were generally big supporters of Huelskamp, Amash and Schweikert. Jones is viewed more as a conservative maverick than a tea party Republican. He has frequently siding against GOP leaders on a range of issues over the years. For example, he voted against the GOP budget because he opposed the changes proposed for Medicare.

Schweikert said it was made clear to him "I should vote for the team more."

"Look, we're walking into the 113th Congress with a smaller majority," Schweikert said. "I would have though the fixation would have been family unity. This isn't the way you start a family meeting."

"The GOP leadership might think they have silenced conservatives, but removing me and others from key committees only confirms our conservative convictions," Huelskamp said in a statement Tuesday. "This is clearly a vindictive move and a sure sign that the GOP establishment cannot handle disagreement."

All four lawmakers had voted against the summer 2011 deal negotiated between Republican leaders and President Barack Obama for extending the government's ability to borrow money in exchange for $1 trillion in spending cuts and the promise of another $1 trillion in reduced deficits. Three of the four, the exception being Schweikert, voted against the Ryan-written GOP budget blueprint that the House passed last March.

Their removal from key committees with jurisdiction over the two issues was viewed by some as a signal to other Republican lawmakers to look favorably on whatever final deal Boehner and Obama put together to avert a "fiscal cliff" combination of automatic tax increases and spending cuts in January.

"If it was intended to be a signal, it's going to be a weak signal because the majority of conservatives are going to do what they think is right based on principle," Jones said.

Amash said he has not been told specifically why he was removed, only that it was not based on his votes and that he should go talk to leadership. He said he voted with the Budget Committee's leadership 95 percent of the time. He said the move is likely to make him more independent in the future.

"Being nice to leadership and playing well with them doesn't pay off," Amash said. "They expect a near total agreement with their approach."

The changes in committee assignments could bring about more discipline from the GOP on high-priority issues next Congress, but conservatives were taking the news as an attack on their priorities.

"As the sun rises this morning we can look at John Boehner, Eric Cantor and Kevin McCarthy and know the opposition is not just across the aisle, but in charge of our own side in the House of Representatives," Erick Erickson wrote on the conservative website, RedState. "All the time and energy I would otherwise have to spend to convince conservatives that these gentlemen would be a problem for the GOP has been spared. They've proven it themselves."