From looking at the approach map, https://www.ukclimbing.com/images/dbpage.php?id=238351 it looks like you could stay on the main footpath until you enter the access land and then cut back east? It's a funny little bit of access land https://binged.it/2GCx2zF
We're looking into this with the Access Team from PDNPA at the moment and a site visit by PDNPA is planned for this Wed, then they will follow up with the landowner. BMC/PDNPA can easily fund a stile over the fence if needed, but this would need landowner permission and agreement to a permissive access route first. Putting signs up without permission could well be counterproductive if the farmers are getting annoyed already. I'd advise steering clear for the time being, whilst we try to work out if anything can be done here. Bear in mind that as is often the case with access issues, this might not be an instant fix.
Doing what you say means crossing walls where no styles exist. I'm pretty sure this led to wall damage in the past (the walls are impressively wonky around there and can easily topple if one crossed them) and is why the farmer enters into any discussion pissed off.
There's a stile over the wall, where the public footpath is, but then there's a very short section which includes pegging a fence, which is on private land as far as I can make out.You could just continue following the public footpath up the hill into the Access land and cut down in an easterly direction to the crag as suggested above.
A quick update: Negotiations are ongoing through the Peak Park access team to hopefully establish an improved permissive access route to prevent damage to walls/fences which is a concern for the landowners. Please stay away from the crag for the time being whilst this discussion takes place to ensure no incidents jeapordise negotiations.
Still working on it James - likely to be a slow burner I think as it will need some charming of the landowners before they agree to anything