"People are tooting their horns, revving their engines. If this was your home, would you want someone to come and disturb your sleep and your life in that way?
"It would be nice if they just left us alone, if we're not doing any direct harm to anyone up here, I don't see why we shouldn't be allowed to stay," he said.
Living rent-free had allowed Callum to drop down his hours working in hospitality and retrain as a joiner, he said.
And now he has a new job, he said he and his partner were looking for a flat.
Callum is one of 107 vehicle-dwellers who Bristol City Council estimates live on the Downs.
Mr Nelson, who founded the Facebook group Protect the Downs, believes living in a van has become a "lifestyle choice".
His group, which has nearly 2,000 members, has called for the council to use its powers to remove vans and those living in them.
Mr Nelson said: "People really feel very strongly about the council's inaction, their permissiveness, their saying 'it's OK to come and trash our parks'.
"People are fed up with that. I don't know if it's council incompetence or whatever."
He wants the council to help those who need it, and move others on.
Between 200 and 300 people – including people who live in vans – gathered on the Downs on Thursday evening for a walk described as a "show of support" for a "safer, cleaner Downs" and to "highlight the impact of council inaction on public wellbeing".
Bristol City Council (BCC) said it was choosing not to move people on as this would simply result in "moving people from one part of the city to the other".
"Every inch of this city is important and special to somebody", said councillor Barry Parsons, who chairs the Homes and Housing Delivery Committee.
The council has previously shut down other encampments – once they were deemed to have had too great an impact on the area.
BBC News was told the current impact on the Downs was assessed to be at a "medium" level, which meant it did not meet the threshold for intervention.
Nevertheless, Mr Parsons said it was "unfair" to claim the council had done nothing, as it had developed a new policy that would be in place "by the end of the year".
"I can understand why people are feeling anxious and frustrated.
"We're trying to do something really new, that hasn't been tried before, here or anywhere else in the country", he said.
There are proposals to provide kerbside "service sites", where people living in vans could dispose of waste and get access to water, as well as plans to open more "meanwhile sites", where people can live in their vans and be provided with basic services.
The council currently has about 60 such pitches, with a new site due to open soon. But there is already a waiting list for spaces.
Privately, several councillors raised concerns about the pace at which change seemed to be happening, acknowledging people needed to see improvements quickly.
And up on the Downs, among people living in vans, and those living beside them, patience is wearing thin.