Fortune will always have its maximum effect when mining ores.
Example: a diamond ore mined with fortune 3, will always give 4 diamonds.
Looting always maximises the amount of extra loot, and increases more the chances for rare drops.
Example1: a looting 2 sword, will always make a creeper drop 2 extra gunpowder, if the default gunpowder range is 0-3, instead of becoming 0-5 (as it would in vanilla), it becomes 2-5.
Example2: each level of looting now gives a 33.4% of obtaining a rare drop, so a looting 3 sword will make you always get mobs' rare drops, like a potato from a zombie or a wither skull from a wither skeleton.
Note that this doesn't mean that every mob will always drop every item they can drop, if the item isn't affected by looting, it won't obviously change anything. For example an elder guardian has 20% of dropping a trim template, but it isn't affected by looting, so a looting 3 sword won't ensure you will get it (in fact it won't make any difference).
Luck of the sea increases the chance of finding a treasure 32% each level, therefore a luck of the sea 3 fishing rod ensures to catch a treasure every time (default % is 5, so 5+32*3 = 101%)
Both fortune and looting should work on modded ores and mobs as long as their loot is affected by fortune/looting.

