The Independent Inquiry into Child Sexual Abuse (IICSA) is examining the extent to which religious groups and local authorities failed children.
Earlier this week a latest victims group - Survivors of Organised and Institutional Abuse (SOIA) - withdrew from the process.
The IICSA said it had "taken on board" a number of issues raised by SOIA.
SOIA said the group had taken the decision to withdraw "with regret" but said the inquiry was "not fit for purpose".
Set up in 2014, the inquiry has been beset by controversy, with three chairwomen stepping down, lawyers quitting and victims losing faith in the process.
'Darkest episodes'One of the victims, Dr Phil Frampton, who grew up in Cornwall, said instead of being at the heart of the inquiry, survivors have been "utterly marginalised".
"This inquiry is not fit for purpose and has never been fit for purpose - we engaged to try to help it be fit for purpose, but it's actually going backwards," Mr Frampton said.
He said the Home Office had a "conflict of interest" and had failed to deal with abuse that had taken place, including in children's homes and approved schools it was responsible for prior to 1970.
"This is one of the darkest episodes in the country's history and if you've got people with conflicts of interest, they're never going to shine a torch into those dark places, for fear they'll see themselves," he said.
Dr Frampton has waived his right to anonymity.
More than 200 victims and survivors are involved in the inquiry, which was launched in 2014 by the then Home Secretary, Theresa May.
SOIA is the second victims' group to withdraw from the inquiry. Last September the Shirley Oaks Survivors Association announced it would be pulling out, having lost faith in an inquiry it believed was not independent.
Decision 'regretted'"The heart of the inquiry is the big institutions who are using taxpayers' money to defend their institutions," Dr Frampton said.
"It is a callous and cold process the inquiry is inflicting on survivors."
The inquiry, chaired by Professor Alexis Jay, involves 13 initial investigations into allegations against local authorities, religious organisations, the armed forces, public and private institutions and people in the public eye.
In a statement, IICSA said it regretted SOIA's decision to withdraw and would welcome the group or individual members back,
It said it had "taken on board" a number of issues raised by SOIA, adding that the "important work" of the inquiry would continue.