As a rule, privy walls don't cave in. In my area they were wood lined and the wood is all but gone. They are usually 2-1/2' x 4' x 6' deep. What is left is vertical hard clay walls that don't cave in. What can cave in however is the dirt and ash that is loosely packed against these walls. For this reason all 4 walls must be defined and you must dig on a level basis all the way to the bottom otherwise a cave in is possible. In other words do not undermine. If you get ahead of yourself that's when the trouble can begin. So put away your chainlink fence and just be safe.
Dan...Great points. For New diggers...and it doesn’t just pertain to privies....when you aren’t sure about the soil...safety often be gauged by performing a simple compressive test. Take a fist full of soil...say the clay type soil from the outer walls. If you can compress the soil and it retains the fist size ball, and doesn’t fall apart, that is trustworthy soil. If it crumbles it's not soil worthy of your trust. That's why digging with an experienced partner is a must. Dig safe so you can dig often.
For the most part, I trust privy walls. I have seen some stone and brickliners that had buldging or leaning walls. In cases like this, you must use your best judgement! I have used 2 x 4's to shore up questionable looking bricks or stones overhead. In some cases, entire walls were removed for safety! I WILL NOT get down in a hole that I don't feel safe in!
Those that I've dug seem quite stable. However, each time I re-enter the pit I check keystones and the general condition of the walls. In the area I am these holes will go down as much as thirty feet! It can feel like the weight of the world is hanging above your head. Dig to the walls and keep a fairly flat bottom (not yours, the pit's).
Safety is paramount. Do not dig alone...
Disclaimer: Gary Owen aka diggermeister shall be held blameless for any injury, death or loss of property etc, due to any and all advice given to anyone in this or any other forum verbal or written form et al.