Configurable Ignore Tag Aliases and Skip Reasons via reqnroll.json

[richard-haslam]

Many teams (well I expect so) need more context than the built-in @ignore tag can provide.
I believe a common pattern is to introduce custom tags such as:

@ignore:bug
@ignore:todo
@ignore:broken
@ignore:flickering

and then use hooks to mark the scenario as skipped with an appropriate reason.
While this works, every team must implement and maintain its own custom hook logic. It would be useful if Reqnroll provided a built-in mechanism for defining ignore tag aliases and their associated skip messages.

For example in reqnroll.json:

{
  "ignoreTags": {
    "ignore:todo": "Scenario not yet implemented",
    "ignore:bug": "Known product defect",
    "ignore:broken": "Broken automation",
    "ignore:flickering": "Flaky test"
  }
}

Which would allow feature authors to simply write:

@ignore:bug
Scenario: User can export a report

and have Reqnroll automatically:

• Treat the scenario as ignored/skipped.
• Generate the appropriate test runner attributes where possible.
• Surface the configured message in test results and reports.
• Keep the implementation runner-agnostic.
• Remove the need for custom hook implementations.

Benefits

• Provides meaningful reasons for ignored scenarios.
• Improves reporting and maintainability.
• Allows teams to define their own taxonomy without framework changes.
• Keeps feature files self-documenting.
• Reduces duplicated custom code across projects.
• Remains fully backward compatible with the existing @ignore tag.

with the implementation details abstracted by Reqnroll.
This would help teams track and report on ignored scenarios while keeping the framework's existing simplicity.

[clrudolphi]

Could this be achieved without changing Reqnroll by using custom @ignoreReason:reasonCode tags?

If Reqnroll did support reason code suffixes to the ignore tag, is the mapping to arbitrary text necessary? Why not simply use the suffix as the reason text? (As in, [Ignore("bug")] )

[richard-haslam]

Yes it can be achieved using custom tags. I thought the idea might have some merits to prevent multiple implementations of roughly the same idea and prided a more consistent approach.

I think the custom text in reqnroll.json would be a nice to have, but agree it would keep it simpler to just use the suffix, and choosing a good suffix should eliminate the need for a more detailed explanation.

[richard-haslam]

I my be jumping the gun, but I've had a little bit of a look at the code base, see that amending TagHelper.cs would easily support mush of this.

public static class TagHelper
    {
        /// <summary>
        /// Checks whether the supplied tags contain the ignore tag
        /// </summary>
        /// <param name="tags">The tags to check.</param>
        /// <returns>A boolean that indicates whether or not the ignore tag is present.</returns>
        public static bool ContainsIgnoreTag(string[]? tags)
        {
            if (tags is null)
            {
                return false;
            }

            for (int i = 0; i < tags.Length; i++)
            {
                if (string.Equals(tags[i], "ignore", System.StringComparison.OrdinalIgnoreCase))
                {
                    return true;
                }
                
                if (tags[i].StartsWith("ignore:", System.StringComparison.OrdinalIgnoreCase))
                {
                    return true;
                }
            }
            return false;
        }

I'm not a developer, so have very much testers experience, so may well be better ways, and I haven't bench marked this solution to make sure it don't have a negative impact on performance using StartsWith.

I would assume that we would want to keep the existing @ignore condition as is for backward compatibility with anyone using custom tags. (Though I would have though anyone starting a tag with 'ignore' probably wants the scenario to be ignored).

I see getting the reason to bubble up to the test output is a little more challenging, But this simple change would allow for filtering in test reports nicely which would be a great start, and to be honest probably enough for me.

[clrudolphi]

Unfortunately, its not quite that simple.
When Reqnroll generates the code-behind for a feature, the generated code contains an invocation of this method. Here is the code generated for the sample "Calculator" project with an @ignore tag placed above the "Add Two Numbers" scenario:

        [global::NUnit.Framework.TestAttribute()]
        [global::NUnit.Framework.DescriptionAttribute("Add two numbers")]
        [global::NUnit.Framework.IgnoreAttribute("Ignored scenario")]
        public async global::System.Threading.Tasks.Task AddTwoNumbers()
        {
            string[] tagsOfScenario = new string[] {
                    "ignore"};
            global::System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new global::System.Collections.Specialized.OrderedDictionary();
            string pickleIndex = "0";
            global::Reqnroll.ScenarioInfo scenarioInfo = new global::Reqnroll.ScenarioInfo("Add two numbers", null, tagsOfScenario, argumentsOfScenario, featureTags, pickleIndex);
            string[] tagsOfRule = ((string[])(null));
            global::Reqnroll.RuleInfo ruleInfo = new global::Reqnroll.RuleInfo("Adding two numbers", null, tagsOfRule);
#line 6
this.ScenarioInitialize(scenarioInfo, ruleInfo);
#line hidden
            if ((global::Reqnroll.TagHelper.ContainsIgnoreTag(scenarioInfo.CombinedTags) || global::Reqnroll.TagHelper.ContainsIgnoreTag(featureTags)))
            {
                await testRunner.SkipScenarioAsync();
            }
            else
            {
                await this.ScenarioStartAsync();
#line 7
 await testRunner.GivenAsync("the first number is 50", ((string)(null)), ((global::Reqnroll.Table)(null)), "Given ");
#line hidden
#line 8
 await testRunner.WhenAsync("the two numbers \'are\' added", ((string)(null)), ((global::Reqnroll.Table)(null)), "When ");
#line hidden
            }
            await this.ScenarioCleanupAsync();
        }
    }

You will see that mid-way through the generated method the static helper method Reqnroll.TagHelper.ContainsIgnoreTag() is called to confirm whether the set of Scenario tags or Feature tags contained the 'ignore' tag. Your proposed change would work when that method gets called.

But also note that our code generator knows whether the test framework in use (in this example, NUnit) supports ignoring tests in the form of an attribute. When the code generator detects an ignore tag, it generates a test framework specific [Ignore] attribute for that method (or class in the case of a feature-level ignore). When that happens, the test execution framework will skip execution of the method entirely. You will see that attribute just above the method declaration.

What you want is for the ignore tag suffix to be the parameter given to the Ignore attribute so that your reason code shows up in the Test Result as to why that scenario got skipped (replacing the generic "Ignored Scenario" that we have today).

The use of the Reqnroll.TagHelper.ContainsIgnoreTag() is there for test execution frameworks that do not support an Ignore type attribute. IIRC, all of our supported test execution frameworks do now support such an attribute. The helper method is a vestige of a time when they did not.

To make this work will require enhancing the generator (and test framework specific generator helpers) to thread the reason code through to spot where the ignore attribute is generated.
If you would like to attempt it, take a look at the IgnoreDecorator and extending the IUnitTestGeneratorProvider interface and changing how the Ignore attributes get generated in each of the test framework specific IUnitTestGeneratorProvider implementations.

[richard-haslam]

I've taken a bit more of a look and seeing it's a bit harder than I first thought. Thanks for the pointers. I'm quite excited to have a go at doing this, would be great to be able to contribute something back.

If I do manage to get this implemented, would this idea be accepted? I don't want to waste peoples time. (My own I'm not so worried about).

[Code-Grump]

I think having a general-purpose extension-point for defining custom ignore tags would be a good feature for us, not least because it's giving users the power to drive behaviour in their domain language.

I would propose a single regular expression option, where if a tag's text matches the expression, it's considered an "ignore" indicator for the test.

If we want to support passing a reason into generated code to be output to the test runner, we'd need to think about options available because the constraints of tags make them quite poor at delivering wordy explanations.

Perhaps a classification system, mapped to a runtime lookup table works here?