Location is revealed through Stolen Land quest, high perception check is just a technicality so that you don't discover it manually.
You can get inside the fort through the gates or through the broken wall passage with (Mobility 20) and/or (Athletics 20) checks.
Passing the Mobility check will get you to the other side and +18xp. Failing will damage the whole party with 3d6 Slashing damage.
Passing the Athletics check will get you to the other side and +18xp. Failing will still get you over to the other side, but 4 hours will pass and alarm will be set off.
(Lawful Good) choice to confront Stag Lord straight away on HIGHLY unfavorable terms: fighting against him, his gang and his lieutenants at the same time, while there are still several bandits in the fort.
Trick the gatekeeper to open the gates with (Bluff 15) (Must have met Akiros Insmort in Thorn Ford) or (Bluff 21) checks. Failing a check just lets you pick another option.
Leave the gates with [Attack] and [Leave] options.
In the fort there boxes that require (Athletics 19) check to remove them and free up some loot. Success grants +18xp. Failure makes you spend 4 hours.