Hi
The 'When' method applies to *all* the previous rules in the chain, so the final condition that checks for an empty string is being applied to all 3 of the rules. You have a couple of options here. Firstly, you can change the behaviour of When by using the ApplyConditionTo parameter:
RuleFor(x => x.ValidatableProperty).Cascade(CascadeMode.StopOnFirstFailure)
.NotNull().When(item => configuration.RequiredField, ApplyConditionTo.CurrentValidator)
.NotEmpty().When(item => configuration.RequiredField, ApplyConditionTo.CurrentValidator)
.Length(5, 10).When(item => !string.IsNullOrWhiteSpace(item.ValidatableProperty), ApplyConditionTo.CurrentValidator);
...and this will make the When clause behave as you want. Personally, when I want rules like this I find it much more readable to put the rules on separate lines:
RuleFor(x => x.ValidatableProperty).NotEmpty().When(x => configuration.RequiredField);
RuleFor(x => x.ValidatableProperty).Length(5, 10).When(item => !string.IsNullOrWhiteSpace(item.ValidatableProperty));
(also note that you don't need to use both NotNull and NotEmpty - NotEmpty also does a null check).
Also, thank you so much for providing a unit test! It makes troubleshooting issues so much easier.