Am I Overdoing It?

Cement parge coat was visably failing in my field stone basement, additionally a hammer tap identies some seemingly hollow spots.

Grabbed an SDS hammer and I'm chipping away, being as careful as I can to leave the more solid lime mortar in place. I plan to repoint and lime wash.

There are some areas with a parge coat that appear to be rock solid and it's more evident when I attempt to chisel them out (the going gets slow)

Should these areas be left or should all the cement be taken to bit? I'm worried I'm going at it to hard