In order for you to be able to steal from a shop, the STOre file's "Can Steal" flag has to be set. The game is hardcoded to allow thievery when this flag is set, and passes control to a script, as Sim pointed out.
The other form of shoplifting, that is controlled by script, is when you try to steal from a container that has the trap flag set and a script associated with the trap. StealFailed(O:Object) is the trigger used. Bioware's scripts check to see if the thief was invisible at the time the steal failed and, if not, summons guards. You'd have to edit those scripts if you want better control of the situation.
That's all I can think of atm. I'll admit this isn't an area I've played with very much.
Edit: updated info a bit.