The reason why it may feel painful to walk barefoot on gravel at first is because your feet need to feel the ground for proprioception and balance, and while wearing shoes, their senses are blunted, so the feet become more sensitive in response.
This becomes natural with time using the feedback from your soles and your eyes naturally processing the ground.
Occasionally, you will step on something pointy even though you looked carefully before placing your foot. If it’s too late to retract a step, quickly shift the weight onto other parts of your foot (heel –> ball, and so on). As you become more conditioned, this will be less of a problem since your soles will be thicker and the (now stronger and more agile) muscles and tendons of your feet will be more adapted to rapid re-configuration. The textures under your feet are a good reminder you are barefoot.