The engine is very poor about giving you text feedback. I would be surprised if glitterdust ever worked on undead sola -- as a "lich" he is immune to level 1 to level 5 spells. However, some spells seem to break through the "immune to spell level X" defense (e.g., Dispel Magic). It's possible that the first time Glitterdust "seemed to work" it just so happened that VampSola was attacking or casting a spell at the same time and broke the invisibility himself. Glitterdust bypasses magic resistance but it does claim to allow a saving throw. If for some reason it got past all of his arcane defenses, you may just have gotten lucky with the save the first time.
Sola casts Spell Immunity: Abjuration so spells like Breach (which is 5th level, and thus wouldn't work anyway) won't work on him. You can either sit that out or try to get around it somehow.