Councillors have approved plans to build six townhouses on a car park at the back of New Shire Hall in Bury St Edmunds.
English Heritage had objected to the proposals saying the new homes would be built too close to an Abbey Wall, a designated monument.
“We consider that it would have a severly adverse impact on the setting of the designated monument, having an overwhelming effect which would lead to substantial harm to its significance,” it said.
But planning officers said the wall would not be damaged and would be made more visible as ivy would be removed.
St Edmundsbury Borough Council’s development control committee defered its decision earlier this month. But following a site visit on Thursday it agreed the development could go ahead.