Which? said regulators must ensure a robust standards system that avoids conflicts of interest and more sharing of responsibilities between local authorities and regulators.
The group added a strategy for enforcement post-Brexit is needed as the UK is likely to take on more responsibility and needs stronger import and export checks.
The Food Standards Agency (FSA) is reviewing the enforcement system and recently completed a trial with the retailer Tesco and pub restaurant chain Mitchells & Butlers.
It looked at if private sector audit data could provide assurance businesses are complying with food law.
The agency said it is looking at ways of moving away from a ‘one-size-fits-all’ approach to regulation that was ‘resource intensive’.
Alex Neill, Which? managing director of home services, said people expect food to be safe but there is clearly still work to be done.
“As we prepare to leave the EU, the government and regulators need to ensure that there is a robust, independent system of enforcement in place to give people confidence that the food they’re eating is hygienic.”
Which? finds varying hygiene standards
Comments come on the back of Which? research that found varying food hygiene standards in the UK, with one in five high or medium risk sites failing to meet requirements.
It found that in 20 local authority areas the chances of someone buying from a food business that isn’t meeting hygiene requirements was as high as one in three.
Hyndburn in Lancashire had the lowest ranking with only 35% of its medium and high risk businesses meeting acceptable hygiene standards.
Erewash in Derbyshire topped the table with a 97% compliance rate.
Which? analysed the 2015/16 Local Authority Enforcement Monitoring System (LAEMS) database of the Food Standards Agency (FSA) to order the 386 local authorities.
It ranked them on the proportion of medium and high risk premises meeting hygiene requirements, the total premises rated for risk and amount of planned interventions (such as inspections or follow up actions) the authorities achieved.
Hygiene risk of a business was based on the type of food, number/type of consumers at risk, method of processing or handling food and confidence in management.
Councils target reduced resources at riskiest businesses
Councillor Simon Blackburn, chair of a Local Government Association (LGA) board, said reduced inspections does not necessarily mean increased safety risk.
"Although it is ultimately the responsibility of food manufacturers, suppliers and retailers to ensure the products they produce or sell comply fully with food safety law, are fit for consumption and pose no risk, councils work extremely hard to maintain and improve food safety standards,” he said.
"Despite the significant funding pressures affecting everyday services, councils are doing everything possible to maintain checks in this area, with several local authorities making significant improvements, as the report highlights.
“Councils know their local areas best and target reduced resources at the riskiest businesses, while national co-ordination through the Food Standards Agency also helps to ensure that areas of the food industry most at risk are generally targeted accordingly.”
Which? also analysed 32 local authorities using data submitted to Food Standards Scotland (FSS).
Edinburgh was bottom and the eighth lowest ranking local authority area in the UK.
The Orkney Islands topped the table and finished seventh in the UK overall.