You can unassign a user from a lead. Once this is done, the Todos associated with the lead will not show up in the user's My Todo List and the user will also stop receiving notifications for that particular lead unless he has followed the lead or his Notifications setting is Follow All Leads.

To unassign user from a lead click on the lead from the Pipeline view,

Click on the Remove option, next to the user's name on the top right corner of the lead details page,