btrfs: fix nodatacow vs compression inode flag conflict check

btrfs: fix nodatacow vs compression inode flag conflict check

Applications expect inode flags to be orthogonal: changes can be combined
or applied separately in any order, as long as each intermediate state is
valid and unchanged flags are left untouched.

Commit f37c563bab42 ("btrfs: add missing check for nocow and compression
inode flags") intended to forbid combining FS_NOCOW_FL with either
FS_NOCOMP_FL or FS_COMPR_FL. The implementation contained a bug and
introduced multiple regressions.

Bug: FS_NOCOW_FL (+C) and FS_NOCOMP_FL (+m) can still be set together in
a single FS_IOC_SETFLAGS call, even though the commit message states this
combination should be rejected.

Regression 1: Switching from +C+m back to -C-m only works if both flag
changes are combined into one ioctl; separate -C and -m calls are
rejected.

Regression 2: Switching between +C-c and -C+c only works if the changes
are split across multiple ioctls; a combined -C+c call is rejected.

Regression 3: Inodes created on kernels before commit f37c563bab42
("btrfs: add missing check for nocow and compression inode flags") with
both COMPR and NOCOW set cannot have any other fsattrs changed on newer
kernels. Even unrelated operations (such as adding +i, or clearing just
one of the compression bits) fail with EINVAL, because the conflict
check rejects the entire ioctl whenever conflicting bits are present,
even if those bits are not being modified. This makes it impossible to
manage older files without first undoing their existing flags.

Fix by:
 * Rewriting the conflict checks so FS_NOCOW_FL cannot be combined with
   FS_NOCOMP_FL or FS_COMPR_FL in any ioctl (fixes the original bug and
   regressions 1-2).
 * Allowing existing conflicting flags to remain if they are not modified
   by the ioctl (fixes regression 3).
 * Moving the check later in the flag-validation sequence so that it
   occurs after handling the long-standing rule that FS_NOCOW_FL changes
   are silently ignored on non-empty files. This preserves the pre-existing
   behavior while still applying the corrected conflict logic.

Also commit the new inode flags to the inode before setting the
compression property, restoring them if the property cannot be set:
property validation rejects compression on nodatacow inodes based on
the inode's current flags, which would otherwise reject a single ioctl
that both clears NOCOW and sets COMPR (regression 2) even with the
conflict check corrected.

Fixes: f37c563bab42 ("btrfs: add missing check for nocow and compression inode flags")
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>