Leading figures in the Caribbean reparatory justice movement have warned that British offers of debt relief and development assistance must not be treated as a substitute for reparations for centuries of slavery.

The pushback follows an announcement on Friday by British prime minister Andy Burnham, who said he wanted to open a structured dialogue on slavery and reparations and was willing to consider an apology, but ruled out paying reparations. In place of compensation, he floated talks on debt resets for smaller Commonwealth countries and pledged £40m ($54m) for critical infrastructure in the Caribbean over the next three years.

Burnham's proposal came as leaders of the 56-member Commonwealth gathered for a private retreat at their summit in Antigua. Ralph Gonsalves, a former prime minister of St Vincent and the Grenadines and a senior figure in the Repair Campaign, said regional governments would probably welcome the offer, but the Caribbean would resist any effort to present debt restructuring or aid as reparation.

He pointed to the 10-point plan drawn up by the Caribbean Community's reparations commission, which calls specifically for "debt cancellation" rather than restructuring, and for "monetary compensation". The plan argues that former colonies' "existing debts are intergenerational and are ultimately and intimately tied to centuries of colonial extraction".

Gonsalves told the Guardian that the areas Burnham and the king highlighted, such as debt relief, education, health and climate infrastructure, all feature in the 10-point plan, but also appear in the United Nations sustainable development goals of 2015, which every country signed. "We have to understand the specificities of reparatory justice as distinct from general sustainable development goals," he said, warning that development assistance remained discretionary on Britain's part.

He also noted that the plan covers matters beyond finance, including ecological repair, psychological repair and possible repatriation for members of the Rastafari faith and others wishing to return to Africa, arrangements he said would carry financial implications and require a separate carve-out.

Bell Ribeiro-Addy, the Labour MP who chairs the House of Commons all-party parliamentary group for Afrikan Reparations, said the issue was fundamentally about "agency". Speaking at an event on Tuesday launching a global reparations caravan that will travel across countries to build engagement with the movement, she said arrangements "absolutely has to be made with those who were affected, because if it is done to us, it is not reparations."

Prof Kingsley Abbott, a human rights lawyer and director of the Institute of Commonwealth Studies at the University of London, echoed that message at the same event, saying harmed communities must have the decisive voice. Drawing on his work with victims of atrocity crimes in Guatemala, Colombia, Nepal and Myanmar, he said survivors everywhere shared one demand: justice, and to be "made whole again".

A Foreign Office spokesperson said: "The prime minister wants to build stronger relationships to renew the relevance of the Commonwealth in a new era. We will do that by facing our past together and focusing on a future where every person living within it has equality, security and hope." The spokesperson added that Britain was already working with Commonwealth partners on private finance for global debt, the climate crisis and economic development, and would continue to do so.