Veterans’ Affairs Minister Matt Keogh has conceded he made a mistake by refusing to meet former defence personnel who are calling for Labor to scrap a cap on allied health service funding, but denied accusations he rolled his eyes at the veterans and refused to shake their hands.

The Coalition of Veterans’ Voices is in Canberra this week urging the government to abandon an annual $5000 cap on allied health services for veterans, introduced in the May budget and set to start next year.

Keogh met the advocacy group on Monday after the Coalition launched a scathing attack on the minister. The allied health services cap has further strained the minister’s relationship with the veteran community after he abandoned plans to scrap funding for the Invictus Games and pull travel assistance funding for the parents of Victoria Cross recipient Cameron Baird.

Former pilot Dr Tim Nelson said he had no faith in the department’s ability to administer the cap and accused Keogh of rolling his eyes during Monday’s meeting.

“It was obvious he did not want to be there,” Nelson told a press conference in Canberra on Tuesday. “He walked into the room, refused to shake our hand, sat down with his arms crossed, and rolled his eyes.”

Former navy clearance diver John Armfield, whose brother died by suicide, said Keogh was too focused on the budget bottom line.

“Minister Keogh forgets he was elected to serve, not self-serve,” Armfield said. “The budget’s obviously playing a role here. He wants the savings to go with that.”

Labor identified the $6.6 billion veterans fund as a target for savings in May, and Finance Minister Katy Gallagher likened it to the NDIS because dodgy doctors were skimming payments meant for retired defence personnel. The budget measures aim to claw back more than $700 million over four years.

Veterans with critical needs can apply for greater funding for services, including occupational therapists, physiotherapists and psychologists once they reach the cap, which Labor says is necessary to prevent overcharging. But veterans have argued that it creates an extra hurdle for those in crisis.

Keogh said on Tuesday he had made a mistake for not agreeing to meet the veterans in the first instance, but rejected Nelson’s characterisation of his behaviour in the meeting.

“I acknowledge that I got it wrong not agreeing to meet with them last week when that initial request came,” he said.

“When I came into that room, I met with them, and they were sitting down. I said, ‘Relax, and I want to hear what you want to tell me’ … and I shook the hands of those people who attended that meeting as we finished that meeting, to be clear.”

Keogh acknowledged the distrust of the department within the veteran community, and said a discussion paper to consult on the changes would be released soon.

He has argued Labor is making it easier for veterans to access allied health services by committing nearly $170 million to increasing fees for providers that had been threatening to cut back services, and said the government was striving to ensure veterans did not pay a gap fee.

Opposition defence industry and personnel spokesman Phillip Thompson, himself a veteran, said the community Keogh represented had lost faith in his leadership.

Thompson has attacked Keogh over the “cruel cuts”, referencing his own mental health struggles and warning that veterans were planning to take their own lives when they hit the annual cap because they swore to never be a burden on their families.

Nationals leader Matt Canavan said allied health services saved lives, and called on the prime minister to meet those affected by the cap.

“Those that do sacrifice for our nation don’t just leave scars on the battlefield. They are left with scars for life, and understandably, they need help … and support from professionals,” he said.

If you or anyone you know needs support, you can contact Lifeline on 131 114, or Beyond Blue on 1300 224 636.