If you're wielding a shield - it is always the shield's def value and the shielding skill that count to your defense (weapon and fighting skill being irrelevant).

If you're wielding only a weapon (without a shield) - the weapon's def value and the corresponding fighting skill count. That is regardless of whether your weapon is one- or two-handed.

If you're wielding neither a weapon nor a shield - fist's def value and the fist fighting skill count.