How can I use Automation for JIRA to create experiment variants?

[Joe Clark]

 Hey Gang!

We are using the Automation for JIRA add-on to send satisfaction survey emails directly from JIRA when an issue is resolved.

I'd like to set up an experiment to A/B test some variants of the email's contents - is it possible to express a configuration with this add-on which would achieve this? Ideally I would like to perform one action for 50% of issues, and another action for the other 50%. My initial idea was that I could just create a condition which does a modulu of the issue key and performs different behavior for even/odd numbers - but I couldn't find a way to do this.

Thanks! 

[andreas · Accepted solution]

Hey Joe!

Interesting use case - I think the short answer is, we don't have something that can do a random 50/50 selection of issues yet.

I've raised https://codebarrel.atlassian.net/browse/AUT-306 for now, but we've got quite a few higher priority items in our backlog at the moment so it will be a while before we can get to this.

The only thing I can think of is to use a regex perhaps to match on odd vs even numbers (e.g. http://stackoverflow.com/questions/691519/regular-expression-to-match-only-odd-or-even-number), however you'd have to ensure that they are mutually exclusive since we don't have else-if yet (https://codebarrel.atlassian.net/browse/AUT-42).

Your rule would look like this:

• Trigger: JQL Search
• Action: Related issues with Current issue
  • Compare Condition: with your regex
  • Send e-mail A
• Action: Related issues with Current issue
  • Compare Condition: with your other mutually exclusive regex
  • Send e-mail B

To be honest I think right now you might be better off using a third party system to send the e-mails that's designed for A/B testing and using an Outgoing webhook action to trigger these e-mails being sent somehow (I don't really know of any that do this though :( ).

Sorry I don't have a better solution yet!

Cheers,
  Andreas

[Joe Clark]

Hey Andreas,

Thanks for the answer! I'll be sure to vote/watch those issues and then complain noisily until they are resolved ;)

I agree we might want to refactor these emails out to an external service to enable deeper experimentation, but for now I'm looking for the quick-and-dirty approach.  Thanks for the suggestion - I'll give the regex approach a try.