A group of fans from a polish heroes community found a following bug:
Terrain: neutral to both units
Attacker:
Hero: 0 attack, 0 defence
Army: 3 ancient behemoths under curse spell
Defender:
Hero: 0 attack, 17 defence
Army: 1 marksman

According to formula: 139 damage. Marksman has 20 defence overall. Ancient behemoths reduce marksman's defence by 80% hence marksman's defence equals 4. Using the formula of damage dealt, behemoths should deal 139 damage.

According to calculator: 139 damage

Reality: 144 damage (as if behemoths had 1 more attack or marksman had 1 less defence)

Solution: although it is stated that ancient behemoths reduce defence by 80%, in reality it is ~80,00000119% because of binary representation of number 80 in memory. Hence marksmen's defence: (100%-80,0000019%)*20 ~= 3.99999962. The value is rounded down and interpreted as 3. So ancient behemoths deal 144 damage points to marksman with overall 3 defence.

Similar kind of bug doesn't apply to golems' resistance to magic.
Similar kind of bug doesn't apply to nixes' ability of decreasing enemy's attack.