Mobs can hide by default. If you want "vanilla sight" for players, set
(server settings -> senses -> use player senses)
to false.

For optifine compatibility, do the following:
1. Disable entity fading (client settings -> entity fading) by setting "min opacity" to 1 and/or "max opacity at" to 0
2. Disable targeting glow (client settings -> HUD -> targeting HUD style) by setting "glow" to false

Dynamic Stealth has an entity-specific full-bypass filter, so in theory any mob that bugs out with DS installed can be "fixed" using that config, but they won't use DS systems anymore.

I've at least done a basic official video guide. It doesn't go through advanced features; just the basics.

Also check out Inforno's BackStab mod; it can be especially useful if you just want backstabbing without all the other stuff that comes with Dynamic Stealth

The sight system uses many factors to decide whether things see each other, and has a config option to bypass it (natural soul sight). Factors include but are not limited to...

1. How far behind the target you are, angle-wise

2. How far you are from the target, distance-wise

3. How bright it is where you're standing

4. Whether you're sneaking

5. Whether you're wearing a mob head that matches the target

6. Armor worn (based on total armor points)

7. Whether you're on fire

8. Invisibility, blindness, glowing...

9. Etc.

The AI is altered in a fairly mod-compatible way, and there is a config option to bypass it (threat bypass). There is also a config option to force the system to detect mobs as passive/non passive if it is detecting them incorrectly (but it detects most of them correctly). Some things the ai does:

1. Searches for its target if it saw a target, or searches in a general area if it is alerted by something

2. Warns other mobs using a customizable filter when it sees a target, takes damage, etc

3. Flees from combat if health is low

4. Rallies if health was low, but is not low anymore, coming back for revenge

5. Gets desperate if cornered while fleeing, and fights back

6. etc...

There are some other features as well, such as...

...at least one new potion effect (soul sight); I may add more later

...some stealth/visibility-related potions

...changes to what can be seen through; players and mobs can see through the exact same things (when trying to see other players/mobs, not when trying to see blocks)

...a HUD showing some data for entities you can see

Most things are very configurable. There are config options to bypass or customize most systems.

It also fixes some things. For example, vanilla mobs (and most modded mobs) can no longer see you through solid diagonal walls.

You can find a configurable graph emulating the base FOV shape (before modifiers are applied) here

Modifiers just take that shape and make it bigger or smaller, but the detection range never goes beyond Distance(Far), so if the shape is increased in size, it's cropped in a circular pattern.

Yes, you can use it in your modpack if you want (MIT license).

A list of my curseforge mods can be found here