When the party meets Imoen, the way back to Brynnlaw is essentially barred, you have to pass through the Spellhold maze and the tests to reach Calahan. That still leaves you with a party of 7 to go through Spellhold, which I always considered the hardest part of Imoen's way back to Athkatla. Though other people might feel different about this...
If the party leaves Brynnlaw with Calahan, the Sahuagin encounter will take place anyway, I suppose? Somehow you'll have to get to the Underdark...
What do you do with Caiya after leaving Brynnlaw with her? If you don't include her as a recruitable NPC, she's doomed nonetheless during the Sahuagin raid; that is, if you don't rewrite the Brynnlaw - City of Caverns - Underdark sequence, or provide her with a similar means of escape as Saemon (Equal Rights for Players! We Demand Inter-Area-Teleport!).
That's what I can think of at the moment...