Figures there's "no" loss to a navvy, outwardly! (5)
I believe the answer is:
fives
But I can't explain it at all! Could you help me understand?
(Other definitions for fives that I've seen before include "horse disease as before" , "bunch capable of causing injury?" , "Small numbers" , "Game similar to squash" , "Ball game" .)