Military retirees are outraged that Congress will start voting Thursday on a budget deal that trims military pensions, calling the move "an egregious breach of faith."

The Military Coalition, some 27 military groups, wrote to leaders in Congress and President Obama late Wednesday about their "strong objection" and "grave concern" over the budget deal.

The deal cuts pension cost of living raises by 1 percentage point for military retirees who aren't disabled and not yet 62 years old. Cost of living hikes are automatic raises intended to keep up with inflation.

The problem is, most military retirees are a lot younger than private sector retirees. They enlist in their 20's and retire in their 40's. Very few stay on till they are 62 -- those who may be lucky enough to escape major injuries at war, or rose to higher echelons in the military system.

When compounded, the 1 percentage point cut could result in much more than a 20% reduction in retiree pensions over 20 years.

Related: 5 things the budget deal doesn't do

The average cut in pension payouts, including compounding interest, for a retiring Army Sergeant first class, would be about $3,700 each year, according to the Military Officers Association of America. Over 20 years, the total losses could balloon to more than $80,000.

"While portrayed as a minor change, a 20% reduction in retired pay and survivor benefit values is a massive cut in military career benefits," wrote groups, including the Air Force Sergeants Association, Iraq & Afghanistan Veterans of America and the Marine Corps League, among others in the letter.

The change is infuriating to military retirees like Army Col. Michael Barron, who retired nearly 4 years ago, after 30 years of service, which included being deployed to both Iraq wars.

"It's not fair at all. I spent a 30 year career in the military. I clearly understood what the (cost of living increases) would be," said Barron, deputy director of government relations for the Military Officers Association of America. "This is the worst kind of example of a shady, backroom deal."

The cut is forcing some to reconsider how much longer they will continue to work with the military.

Rebekah Sanderlin's husband is two years away from hitting his 20 year retirement mark with the Army and she's wondering if its worth it. He's served in Afghanistan four times, among other places, and has many injuries.

"The war has been very hard on our family," said Sanderlin, a writer. "We'd like to stay in, but it seems stupid to give more time to a government that goes back on their word."

Military groups say the cut is particularly unfair because the changes will affect those who have already put in their years of service.

"To tax the very men and women who have sacrificed and served more than others is simply a foul," the letter stated.

Related: A new normal for government retirees

Washington leaders, and House Republicans, in particular, have been worried about the cost of military retiree benefits.

In 2012, the Pentagon spent $52.4 billion on 2.3 million military retirees and survivors, a cost that is expected to rise over the next few decades, according to the Department of Defense Office of the Actuary.

House budget chief Rep. Paul Ryan's website states that military retirement "provides an exceptionally generous benefit, often providing 40 years of pension payments in return for 20 years of service," as it explains why benefits should be trimmed.

"Current levels of military compensation are incompatible with the overall demands on the defense budget," according to a House Committee on the Budget Report.

Military groups say they're open to reforms, but they'd like such changes to go through the normal legislative process that allows time to review and "assess any recommendations that could significantly impact retention and readiness."

Barron said groups like his were "blindsided" by the cuts to military pensions.

The cut ends up saving the budget $6 billion, according to the Congressional Budget Office. Congress would also make newly hired civilian federal workers contribute 1.3% more of their paychecks to pensions if the budget deal becomes law.

The Department of Defense wouldn't comment on the Military Coalition letter and pointed to a statement by Defense Secretary Chuck Hagel saying the budget deal provides greater "budget certainty," while reducing the impact of massive cuts from so-called sequester.

Correction: An earlier version of this article incorrectly characterized the cost of living cut as 1%. In fact, it is a cut of 1 percentage point.