Perhaps try this?

gogs/gogs#535 (comment)

"I discovered the solution. After deleting, you must go back into the repo, select Settings > Manage Access > Manage, then delete again, using the same process."

Also, check out the following post which explains that this issue may be related to caching:

desktop/desktop#8586

Steps you could take:

• Delete browser cache again
• Restart the GitHub Desktop app (if you have it)
• Restart your laptop / computer
• Try logging into GitHub in a private browser window, and see if the issue persists.