Chipped plaster in pool?

There are several rough spots on the bottom of my pool. I ume that this means the plaster has chipped off. Because some of these spots are at the deep end of the pool, I would have to drain the whole thing to repair them, and draining the whole pool is a very expensive proposition. Do I need to fix these spots? Will it cause permanant damange if I leave it as is? FYI- I'm not losing any pool water.