When you're in your garrison, the special ability button is displayed right in the middle of the screen. Of course, you can drag this ability to an empty slot on your toolbar to hide it, but then you waste a toolbar slot since you no longer need this button anymore.

This addon lets you hide the Garrison button without wasting a toolbar slot!

You can disable this anytime in game by typing: /run ZoneAbilityFrame:Show()