Losing 5 pounds would not be so noticeable, to the point where people would comment on it. Neither would gain 5 pounds.
Losing or gaining 8–10 pounds would become noticeable, but don't think that just because people don't mention it, that it's not noticeable.
Many times, when you see someone often, they won't notice a stable loss or gain of weight, because they see you so much. It’s the same as looking in the mirror, you may not notice that you're losing or gaining weight because you see yourself every day, so you're used to seeing the stable decline/incline of weight.
My suggestion; You don't need to lose/gain any weight at all if you're within a healthy BMI range, but if it makes you feel more comfortable with yourself, you should allow yourself to lose/gain as long as you remain in a healthy BMI range.
If your goal is for others to notice the loss, the suggestion for the amount of weight is between 8–10 pounds.