The demonstrators gathered in downtown Durham on Friday and marched for several blocks with signs such as "Black Lives Matter" and "We Will Not Be Intimidated" as well as "History will not repeat."
The counter-protesters also burned white supremacist and confederate flags.
Rumors of a white supremacist rally circulated on social media Friday and activists announced that a counter-protest would be held in response to the demonstration.
The sheriff in Durham said in a statement that no gathering of white supremacists was apparent by afternoon but he was investigating the rumors.
On Monday, protesters tore down a Confederate statue in Durham. Efforts to remove such statues around the country have increased since the violent rally last week in Charlottesville, Virginia, where one person was killed and 19 others were injured.
The white supremacists, neo-Nazis and KKK members participating at the "Unite the Right" event in Charlottesville were protesting against the removal of Confederate monuments and memorials from public spaces, specifically the Robert E. Lee statue in Emancipation Park in Charlottesville.
Supporters of Confederate memorials argue they represent an important part of history, while opponents view them as symbols of hate and racism as well as an affront to African-Americans.
On Friday, the mayor of Charlottesville called for an emergency meeting of state lawmakers to allow the city to swiftly remove the statue.
The US Conference of Mayors and the Anti-Defamation League announced a partnership to “combat hate” in wake of the violent protests in Virginia last weekend.
More than 200 mayors have joined the compact including those from New York, Los Angeles, Chicago, Philadelphia, Houston, Dallas, Baltimore, Atlanta and Washington, DC.
“We all have to speak with absolute moral clarity on this issue, and so we need to be clear in this country that this is a time of choosing,” Mitch Landrieu, mayor of New Orleans, Louisiana, and current president of the US Conference of Mayors.