fix: use adjoint matrix when computing 3x3 inverse

fix: use adjoint matrix when computing 3x3 inverse #18605