Adding or removing contributors to a GitHub organization should be straightforward through the "People" section of the organization's settings, but it sounds like you're encountering some issues. Here's a basic guide to help you:

To Add People:

1. Navigate to your organization's GitHub page.
2. Click on "People" in the organization's settings.
3. Use the "Invite member" button to invite people by typing their username.

To Remove People:

1. Again, go to "People" in the organization's settings.
2. Find the member you want to remove, and you should see options to remove or change their role next to their name.

Remember, you need to be an owner of the organization to manage members effectively. If you are a member with limited privileges, you might not be able to add or remove contributors. Also, organizational rules might affect your ability to make these changes.

If you're still having trouble, it might help to see a screenshot of what you're seeing. This can provide more context to diagnose the problem.

This is a picture of my organization that may help you out:

For more detailed instructions and guidelines, you can refer to GitHub's official documentation on adding people to your organization and about organization membership.

Hope this helps!