You can delete an existing contact by clicking on Contacts under the More dropdown,

Find the contact from the list and click Delete next to that contact,

Deleting a contact will also remove it from all the leads where it was previously added.