Skip navigation
All Places > Champion Program > Blog
1 2 3 4 5 Previous Next

Champion Program

124 posts

Hi, everyone!


This was the 12th and FINAL episode of #Krewechats for 2016! We'll be back next year with a full schedule of content to share and discuss, but in the meantime, check out the most recent episode here:


#KreweChats Episode 12: Changes You Should Know About! (2016 Year-end Edition) - YouTube


Last week, we discussed basic changes to the Marketing landscape that just about everyone needs to be aware of. If you don't know what CASL is and you're emailing people in Canada, the first half of this episode is for you! Beyond that, we got into the upcoming deadline regarding the Marketo Lead Management package for Salesforce CRM, as well as the "Fall" Release (that came out on the 9th day of December, in the two thousandth and sixteenth year of our Lord).


Special thanks to Geoff Krajeski, Jenn DiMaria, Ande Kempf, Dory Viscogliosi, Juli James, Rachel Egan, and Sydney Mulligan. Almost a full Krewe for the first time ever!



You already know that Marketo isn't going to send the same email to the same person via the Customer Engagement Program. BUT. . . those aren't the only people that you want to exclude when you are running a nurture program.  You might be promoting a piece of content across 5 different channels and using multiple different emails.

What if someone attends a Webinar on “How to Snag Cool Marketo Swag at Summit” and you want to offer the recorded webinar in your nurture stream, how do you make sure that the person who attended that webinar doesn’t get that offer AGAIN.  Here are 4 steps to make sure that doesn’t happen.


Here's what we are going to build:

Screen Shot 2016-12-11 at 2.36.40 PM.png


Step 1:  Set up a Content Program for each piece of content that you are also promoting via nurtures. 


In this program "Content - Webinar Cool Swag", anyone who attended the live webinar, or watched the recorded webinar already needs to be added as a member in order to exclude them from receiving a nurture email with this webinar as the offer.   This would work the same way if it was a white paper.  Anyone from any channel that has downloaded a specific white paper would live in this program.  This program can be operational or not depending on your reporting needs.  But the important thing is that everyone you want to exclude for this specific nurture email resides in this program. 


Step 2:  Set up a Nurture Library


This step isn’t actually required to make the above happened but it’s more of a best practice and helps you keep things organized.  In this operational program, you can keep all your nurture emails here so that if you choose to use the same email (by email ID) in multiple streams or multiple programs, you will guarantee that folks definitely won’t get the same email twice. 


Step 3:  Set up your Smart Campaign to send the Nurture email from the Content Program


Hold up. What!?!  Yup that’s right.  Create a Smart Campaign (ie Nurture Send) in the Content Program, the same program where the members reside where you want to exclude folks from receiving the email.  This is the part where marketo logic just got flipped on it’s head.  Just stay with me.  Smart List must be Member of Engagement Program = True, Nurture Program Name. Add another filter to exclude Member of Program = False, Content - Webinar Cool Swag.  In the flow step, Send Email - email is choose your nurture email from the Nurture library.  You do not have to turn anything on.  You do not have to schedule anything. 


Screen Shot 2016-12-11 at 2.37.21 PM.png

Go to the Nurture Stream where you want this email to go out and click Add, select program, select the campaign “Nurture Send” and voila that’s it. 

Screen Shot 2016-12-11 at 2.35.45 PM.png

What will happen is anyone who is a part of the Content - Webinar Cool Swag program will not receive Nurture Email 1 offering this very cool webinar, but will get the next email in the stream all is good. 


You may be tempted to drag other filters in the smart campaign smart list. Resist the urge.   They will not work.*  When you use other filters, member of list not in XYZ (and these folks are not members of the Content - Webinar Cool Swag program,  those people who you want to exclude will be excluded from the nurture email, BUT they will not get another email.  They will be stuck in email nurture purgatory.  You have to turn the cadence off then back on for them to leave purgatory.  There have been lots of articles written on this. 


Step 4: Test the Nurture Program


Screen Shot 2016-12-11 at 2.35.57 PM.png


I mean really test it, not with just the test cadence (shown above) where it just sends out an email like send sample email.  Here’s a way that I came up with to quickly test whether the right people are getting the right email and being excluded from the right nurtures. And you don’t have to wait for Marketo to send out a cast.  The shortest amount of time that the program will cast is 24 hours.  I don’t know about you but I don’t have that kind of time to just sit and wait around. 


Say for example you have 4 nurture emails. 

Create a test list of leads that are new to the database.  This ensures that there aren’t any gremlins that are going to mess up your testing.


Screen Shot 2016-12-11 at 2.42.04 PM.png


I have 5 test leads.  The first will proceed as normal through the flows of all 4 emails.

Each of the others, I will add 1 lead to each content program for them to be excluded.

So Jess1 will be in the content program for Nurture Email 1 so it will not get Email 1

Jess 2 will be added as a member of the content program for Nurture Email 2 so it will not get Email 2 and so forth.  The chart delineates which email will be sent to whom and it what order. 


Upload these 5 test leads.

Add them to the appropriate content program.

Add all 5 leads to the engagement program and nurture stream.


When you are ready to test, set the first cast for an immediate cast.  (Make sure you are not sending to actual real people.)  Wait for the emails. Once you’ve received the emails, you can go back in and set a new time for the first cast (ie within the next 15 minutes) and let her roll and voila you can test your nurtures pretty quickly.  


And there you have it. 

Hi Marketo Community!


Last week we hosted our latest episode of #KreweChats, which can be seen again here: #KreweChats Episode 11: Email Editor 2.0 & Audit Trail - YouTube!  I can’t quite believe we’re already up to 11 episodes.  So, for this one we looked back at some of the most sought after releases of 2016.  These were Email Editor 2.0 and Audit Trail. 


On the chat was Joe Reitz, Rachel Egan, Geoff Krajeski, Ande Kempf, Dory Viscogliosi and Myself.  It included a couple of Live Screen Shares (Thank you Joe!) to show what Email Editor 2.0 and Audit Trail look like within an instance.  We discussed the benefits and disadvantages of both new features and looked at why they were needed and how they could be improved for the next release – we wouldn’t be Champions if we didn’t always have ideas for improvements!


Hope you enjoy watching this eposide and we look forward to seeing you for our last epsodie of the year on 12/16/16 @ 3:30pm ET.  On this episode we will take a look back at 2016 and see how the world of Marketing, Marketo and KreweChats has grown.  Come along and get Merry with us!






#KreweChats Episode 11: Email Editor 2.0 & Audit Trail - YouTube

Hey #MKTGNation!


Thanks for your interest in the Marketo-Fu series thus far! Personally/selfishly, having a down-to-earth video link I can share has saved me so much time in training clients or further edifying other Fathomers & Marketo-enthusiast friends. I'm also up to like $14 in ad revenue between this and #KreweChats, so I'm well on my way to my personal financial splurge goal. #LeicaLyfe #IMayRetireSoon


I mentioned before that there would be an intermediate/advanced track once I filled in the beginner track a bit. I've done that! So for your viewing pleasure, here's the first 10 episodes of Marketo-Fu for Intermediate/Advanced Users:


Marketo-Fu (Intermediate) - YouTube


ICYMI: Marketo-Fu (pronounced like "kung-fu") is an on-going how-to series delivered on a semi-daily basis via YouTube live. The purpose is to help you make your Marketo-Fu strong, so you can impress your management, get a raise, and have an awesome Bruce Lee-esque montage (hush, you. I'm funny, dang it...). It's super informal and genuine, and best of all you get to see me forget how to Marketo sometimes .


^Spoiler: anyone that tells you they don't have anything left to learn is straight-up lying to you.


Please feel free to post additional topics you'd like to see on this channel below, and keep the conversation going here!


PS: Once again, YUGE shoutout to Sanford Whiteman for his massive technical expertise!

Here's a piece I recently wrote up to share with some colleagues who felt that they needed to add a lot of custom fields to our Marketo DB for one particular product line in one particular region.  The problem here is that we have so many product lines in so many regions that giving this access to one opens up the floodgates to all.  The problem is that I'm not a fan of sparsely populated data fields - fields that have only a small records who will have something other than the default value.


If the field can't work for everyone everywhere, I'm not interested in adding it to the database.  With that, here's my justification for using static lists instead of custom fields.



Using Flex Fields and Static Lists to act as custom fields


This document will show how to use temporary fields, “Flex Fields”, on forms in place of creating custom fields and then store the data in either Static Lists of note fields for permanent, long term storage and usage.


Flex Fields: We allow four flex fields available for use by all Marketo users. These are called Flex Field 1, Flex Field 2, Flex Field 3 and Flex Field 4. These can be used to store any type of information temporarily – for a half hour – which is enough time to move the data to either a Note field or a Static List for segmentation purposes.


Types of Data: there are 3 main types of questions and data:

  • Finite Choices: These are questions that have a limited number of answers such as Yes / No or Multiple Choice
  • Scalar Data: These are data that use a number to represent a scale. Examples would be number of users, ratings, etc. If this information can be grouped – such as “From 1 to 100” or “Less than 100” or “More than 9000” it can be stored in lists for those ranges: 1-100, 101-500, 501-1000, etc. These ranges can be unique for each form and product line.
  • Free Form Text: This is data that the form respondent supplies by typing in an answer. This data is not standardized, may contain typing errors and cannot be used for reporting.


Fields vs. Lists:

Fields are points of data that exist on every record in every region in the database. Fields tend to be limited to data points applicable to every lead in every region or at least enough leads to compromise a significant amount, or are used by numerous product lines and regions.  While information like Name, Email Address, Phone Number, Company Name and the like are applicable to every lead some information like Number of Operating Theaters, Number of Births, Number of Doctors, Number of Beds in ICU are not used by all product lines but are used by several product lines in several regions.


Static Lists are lists to which we can add records to indicate having a response to a custom question. If we have a question that cannot be answered by an existing field, and that answer is multiple choice or Yes / No, we can write that record to a list to show interest.


Most of the existing roles can create static lists and add or remove members from the list. In combination with the Flex Fields, thousands of points of data can be collected without adding a custom field.


If the data is more free-form in nature, that is to say it’s information that the customer provides that is not their personal contact information, or it’s information that then needs to be passed to SalesForce, that information can be written to a note field.


Presentation on Forms:


When creating a form, add the custom field and then change the label to have the field appear like a custom question. Below is an example of a form used by Healthcare Digital for collecting specific product interest by using a Flex Field and assigning a couple of choices the form respondent can pick from.


Both of these questions below tie to Flex Fields (Values shown for example)




Storing in Static Lists:


Within Marketo, there is a Smart Campaign flow step called “Add to List.” Within the flow step, there is a button on the Top Right called “Add Choice” which acts as IF/THEN type of logic. Putting in multiple choices in a single step means that Marketo will look for the first time the conditions are true and then abandon the rest of the choices. Alternatively, you can add multiple flow steps.  The example below is from the Product Preference project in place for Europe, where information that maps to a single, temporary Marketo fields needs to be stored permanently. The information is delivered with multiple values at once, which then need to be separated into lists.  It’s worth noting that almost 100 static lists are used to track interest in products for Core Imaging alone.


Here is an example of using a Flex Field to drive a product interest into a Static List:

Storing in Notes:


When the information needs to be passed to SalesForce, the only mechanism is to either put in a pre-existing field or into a note field. This also works for data that cannot be used for segmentation, such as self-entered information.  An example I like to use is the number of says someone could type in “Saint Joseph’s Hospital”: St. Joseph, St Joseph’s, St. Joe’s, SJHC, etc. All of these appear as completely discrete values to a computer which would have no way to group them. This self-entered information is best put into a note field.


Example: In this Smart Campaign, we build a Form Lead Note.  We’ve standardized how we use the Flex Fields on these forms.


Since the entire new value isn’t visible, here are the New Values:


Flex Field 1 is always used for Product Interest on certain Healthcare Digital forms:

Lead Notes: {{ question}}; Facility Type: {{lead.Facility Type}}; Product interest: {{lead.Flex Field 1}}


Flex Field 4 is always used for miscellaneous purposes on Healthcare Digital forms:

Lead Notes: {{ question}}; Facility Type: {{lead.Facility Type}}; Product interest: {{lead.Flex Field 1}}; {{lead.Flex Field 4}}

These values then carry over to SFDC where they appear in the Notes field.


Flex Fields used in scoring campaigns


Sometimes for product specific scoring, we want to ask questions that are very unique to an individual product. An example here is for our Cardiology product line, which asks which kinds of labs the hospital has. We store this information in Flex Field 3 from the form. When the form is filled out, this scoring workflow is triggered:

This scoring program will only start if the person indicates they have one of the above mentioned Cardiology Lab environments, which is asked on the form.


Using Static Lists for Segmentation


Now that you’ve seen examples how to translate form questions to Static Lists, the next step is to show you how those lists can be used like custom fields.


A Best Practice for segmentation lists is to create a program in an Operational Programs folder. This is an Operational Type program used only to house your lists.  These can also be stored in the Lead Database, but to make things simple we’ll show how these create what we in Healthcare Digital call “Core Lists.”

Here you’ll see the Core Lists we use for Cardiology. In here are two types of lists: Static Lists and Smart Lists. A static list is basically a list of names that you decide who is on or off.  A Smart List is more like a query: you use logic to determine if someone should be a member or not.


Static Lists give you ultimate control over who is on the lists but need to have Smart Campaigns – workflows – to add or remove names.


Smart Lists constantly update themselves as records qualify or disqualify. Smart lists can use static lists as inclusion or exclusion criteria.


In this example, we have created lists for our various Cardiology products. Lists marked as “IB” are current customers. CCE is one version of our product, CCI, CCW and DMS are other versions. Because this product line is very volatile – new products are constantly added, product names are subject to change and product lines can be retired – having these products as unique fields is not scalable. While the product is called DMS today it could be called ABC or something completely different tomorrow. Once a field is created in Marketo and is populated with data you cannot change the field name. It is also very labor intensive to remove fields.



Building a Segment


In this next graphic, you’ll see how a Smart List uses several Static Lists for segmentation. The purpose of this list is to dynamically build a list of people at hospitals who own our product, who would be interested in hearing about our Perinatal product line.


We know that hospitals have numerous specialties that are of no interest to each other. If we were to generate a list of all hospitals who owned a product and then sent to all records working at that hospital our Unsubscribe numbers would quickly elevate. Why would a neurosurgeon or a proctologist care about Perinatal software? Likewise, as a software company, we know that executives, business analysts and IT personel will sign up for information on this product line while they are evaluating or installing the product but they have no interest in communications meant for the end users.


This Smart List combines all people we know who are listed as being tied to one of the 12 Perinatal products and then filters them by job title. Here were are looking for 107 values that we feel are unique to people who work in women’s health: maternity, labor and delivery, infant care, obstetrics or gynecology, etc.



If these product interests were separate fields we would have to create a very complex list saying (1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13) and 14 and 15 and 16 and 17 as each product would be a separate line. From experience, the concept of “OR” and parenthesis is hard to explain to an average Marketo user while just putting all of the lists into a single box is much easier. Every time we add in a product line we would have to add it to this list and then manually rewrite the Advanced Filters as points 14-17 would need to be incremented by the number of new fields added in parentheses.


Living From Experience:


Prior to 2015, Healthcare Digital had our own instance of Marketo which had been in place since 2011. When it started, the thought was that unique fields would be used as needed and every product interest could have a unique field to show interest.


Over 5 years, that ended up created 121 fields. Many of these are for products that never ended up being real – we prepared for them but they were canceled. Many of these product ended up being retired. Others had their names changed and we were stuck referring to them by an old name. We also made fields to show is someone owned the product or was merely interested in it. This has been difficult to work with.





























































































































When we migrated to the Global instance we made a decision to never make this mistake again. This wasn’t scalable, was hard to support and very hard to train when product names change or retire. All of these issues go away by using lists.



It is my experience that there are very few reasons for adding in new fields, especially product specific or region specific fields. Having given this much thought, testing and experience and conferring with other Marketo Experts, Champions and Consultants, the consensus is that static lists work very well for preserving information that is Boolean in nature, such as a product interest.


As a Marketo admin, my goal isn’t to reject all requests but to work with the regional users and show them how results can be achieved without disrupting the database by adding fields. Along with the other admins we carefully consider and discuss proposals for new fields and judge it on the following criteria:


  • Can the same results be achieved in any other way?
  • Would the requested field serve any other region or product line?
  • Is the field name going to persist for years?
  • Is the field relevant to a large portion of the database? (At least 15%)


If we can achieve results without adding a new field and if data is only relevant to one product line or region then we are not likely to add in the new field.


The reason for not adding new fields is simply scale. If we do it for one product in one region we have to offer the same opportunity for every other product in every region. This is not scalable and would be very difficult to administer. There are too many products in too many regions and in the end only a very small number of records would utilize the field.

Hi Marketing Nation!


Thanks for joining us for another episode of Krewe Chats. This week, we had Dory Viscogliosi, Geoff Krajeski (+his kids if you pay verryyy close attention ) and Jenn DiMaria together to talk about Attribution. The million dollar question when proving the value of anything you’re doing in Marketing: what’s the ROI? As a marketer, this can be a daunting question...UNLESS you have proper attribution reporting setup in Marketo. Listen as we are explore some of the ways attribution can help you prove your value as a marketer and the value marketing brings to your organization’s bottom line.


#KreweChats Episode 10: Attribution - YouTube


Well, uh, that's all. I love attribution. We like you guys. Bye!

This is the first in a series I'm titling "Big Marketo." 


Big Marketo Defined

Big Marketo is about how large companies think about Marketing Automation, the challenges we face and how we think through ideas that are scalable for different skill levels.  Big Marketo isn't just for large companies, it's for anyone who manages more than one or two users of Marketo.  Big Marketo is, in essence, tips and tricks that help us keep this sane in an insane world of ideas.



Having worked for seven years using Adobe Campaign (formerly Neolane) for a medium sized company I came over to GE Healthcare Digital in 2014. Since joining I've become a recognized expert across all of GE in the Marketing Automation arena and ascended to the top level at GE Healthcare to help lead and shape the Marketing Automation initiative. I've redesigned how programs are run in a way that's scalable and simple. I've utilized the power of Tokens, Landing Pages, Script and Campaigns to take the burden off of the end user.  I've written user roles in a way that's sensible, logical and safe. Is it perfect? No. Is it working? Yes.


I currently admin an instance of Marketo with about 160 users and growing. We have users on almost every continent and region and country- North America, Asia, India, Africa, Australia, South America, Europe, Middle East, Far East, Central America, U.S., Canada, etc. Just a couple of months ago it was out of control. We had over 19 people with Admin access, some of whom were consultants, many of whom had no experience or training. People were free to make major changes without knowing the downstream impact.  We also had about 10 different user roles that no one truly understood. Some of the roles were specific to one or two people, others were assigned to people who also had Admin access.


Prior to me joining the Global team, there was a completely different team in charge and all of them had left the company, leaving a large knowledge gap of why it had gotten so out of hand. Assigning blame wasn't the priority, fixing things was. This was going to be painful, we were going to reduce a lot of people's access and take away functionality.  We had seen too many things go wrong and we needed to make this error proof.


So why start with User Roles?

I've thought about it and I'm starting with user roles because I believe from here it will set the table for the other topics.  Everything starts with set-up and talent. You need to know who you have and what there skills are and channel them into ways that doesn't force them to take on responsibilities out of scope.  My next topic will be Program Creation, and that will really help you see how these roles work together. We'll also get into Workspaces and how to utilize them to make your instance safe. But I think the first thing to think about is User Roles, what you should allow users to do, who does what and why and how to make them all work together efficiently.


Let's start at the beginning....what are the core functionalities of Marketo?

Marketo, broken down, has a couple of key areas that require different skill sets. Let's break it down:

Admin -

These are Marketo experts who really know all the ins and outs of Marketo. The wield the highest level of power and use it the least. These are the people who need to make sure Marketo changes as little as possible and see the impact of people's mistakes. Admins real responsibility lies is creating new users, assigning roles, training other users and preserving the integrity and structure of Marketo.


Data Analysis, Scoring & Reporting

Marketo captures and creates data which are used to create reports and models for end users. Marketo has several ways to report data from smart lists to canned reports to their Revenue Cycle tools. A core functionality of your Marketo team should be business analysts who understand databases and data modeling and how to perform testing for scoring and lead lifecycle. Data is captured not only from Marketo but also from any linked CRM or other tool.  Here is where a keen eye for which data is meaningful to have in Marketo vs. which is not is crucial as it will help your instance operate proficiently and not get backed up receiving and transmitting useless data.


Trigger Based Automation

Marketo records a lot of data and is capable of recognizing a lot of events happening. The trick here is to know which are the significant few triggers that your business needs. Knowing how your customer base interacts with your programs is important in knowing which triggers you want to use, as is understanding what your sales and marketing people want to know about. For some, a web page visit might be an important event to note while for others this trigger could overwhelm the system if the website is highly trafficked.


Web / Email Design

Knowing how to design Marketo pages that integrate seamlessly with your website lies in the hands of graphic designers and web designers. Knowing how to model CSS and centralize on it to minimize updates is highly useful for big companies as the web layouts and looks can change quickly.  You want to do this right the first time so 4-5 years down the line you don't end up with hundreds of pages that need individual updates.

You also need people who understand how to design Emails. These use different rules than Web Pages and if you don't know what they are you need someone who does. The email design tool in Marketo is not indicative of how the email will look in Outlook, Yahoo, Gmail, iOS, Android,, mobile, tablet, PC, etc. If you think "All my customers are B2B so they all use Outlook," you're wrong.  If you're doing your layouts on a P.C. and not optimizing for mobile, you're old fashioned. You need to make sure you have a designer that knows how to build and test emails for best performance, as well as how to A/B test.


Web Developers

On top of design, knowing scripting will add immediate value to your Marketo initiatives. If you want to break the mold and show that your website is modern you need to know a little scripting, whether it's JavaScript, jQuery, or PHP.  It's not difficult to learn the basics but you need to be able to incorporate scripting if you really want to get the most out of your Marketo pages.


The Roles and their responsibilities

Your user base

Chances are your user base falls into one of two categories: I'll call the first "Field Marketing," people who are responsible for creating the message, identifying the streams and ways to promote / spread the message, and; "Marketing Operations," your technical folks who know how to make the automation happen. Both of these roles serve unique purposes and should have Marketo roles in scope with their work. With that in mind, here is how GE Healthcare crafted user roles for our instance of Marketo. We'll start with the lower role and work up.


Content Editor:

This is where the majority of your "Field Marketing" team will be. They have ability to clone programs, create new emails or landing pages, update content but not approve unapproved asset and launch / activate campaigns. They have access to Lead Database, Marketing Activities and Analytics but cannot change leads attributes. The main job of the Content Editor is to own the actual content, update it when necessary and create new programs based on templates without the ability to launch unsupervised.

In the next part in this series you'll understand how a Content Editor can do their job proficiently by utilizing only tokens and never needing to know the Landing Page Editor or the Email Editor. For now, just think that these are people who will generate new ideas, forge partnerships with external channels, work with product marketing and sales to identify opportunities to create new assets and then update the customer facing content when necessary.


Required Skillset:

Basic Marketo knowledge and experience using a WYSIWYG editor.



  • Create HTML content and get approval
  • Coordinate with Web team for downloadables
  • Request / schedule campaign on Marketing calendar
  • Lead import list properly requested
  • Clone Marketo program
  • Load content into Marketo asset
  • Update all relevant tokens
  • Prepare workflows to best of ability
  • Notify Marketing Approver when it is ready to launch
  • Work with CE Marketing Approver on launch checklist
  • Ultimately responsible for content and program


Marketing Approver

The next level of skill is your Marketing Approver. This roles has Content Editor plus the ability to approve assets and launch / activate campaigns. Has access to Lead Database, Marketing Activities and Analytics. Is typically restricted to Operation team users with Marketo training. The main reason for a Marketing Approver is that no one should be able to approve their own programs without another set of eyes on them; and prior to launch, an expert in the area of Marketing Automation should review the program to insure it adheres to local laws, company governance rules and it will work as expected.  This role has perhaps the most responsibility as they are responsible for checking every link, every email, every page, every workflow and list prior to hitting Send.


Required Skillsets:

Marketing Automation experience, knowledge of personal data privacy laws for their region, proven comprehension of region governance, workflows, mid-level graphic design (HTML / CSS for webpages and emails).


  • Program set-up
    • Program is in the correct folder
    • Correct program type used: default, event or email send
    • Correct channel is selected
    • Program contains only one channel (i.e. an email pointing to a landing page for a download is two separate programs: email and gated asset)
    • Forms are reused whenever possible
    • Workflows are centralized whenever possible
    • Program cost and period properly set and updated upon request
    • For Webinar, program is linked to the correct webinar platform (WebEx) and event
  • Email
    • Templates are current and active
    • Emails display as intended in different browsers / email clients (specifically test Outlook, Gmail and iPhone / Android)
    • Links are active, valid and are not personalized (if they lead to third party sites)
    • Operational / Non operational email type used correctly
    • Spelling / Grammar is acceptable (if proficient in that language)
    • Text Only version updated and formatted
    • Default values entered for personalization tokens
    • Ultimately responsible that program was executed upon approval from requestor & that protocols were followed
  • Landing Page
    • Templates are current and active
    • Pages display as intended in Chrome, IE, FireFox and mobile (Safari / Chrome)
    • Links are active, valid and are not personalized (if they lead to third party sites)
    • Correct form used and hidden fields updated
    • Head tags populated (content is responsibility of Content Editor user)
    • Spelling / Grammar is acceptable (if proficient in that language)
    • Token codes are not visible on rendered pages
    • Upon form fill, expected page is reached
  • List
    • List criteria matches persona entered on request form
    • For email, list is under 10,000 recipients
    • Unsubscribes, invalid emails, hard-bounce replies, blacklisted email addresses are removed
    • List contains people for whom the information is relevant and expected
  • Workflow
    • Proper list is used / created in Smart Campaign
    • Flow steps are tested to ensure expected results are achieved
    • Values / tokens used in flow steps are correct for the product and program
    • Note field updates are formatted properly and append to previous notes and do not wipe out data
    • Activities written to CRM fields are formatted properly
    • Leads score as expected and can PQL
    • Program Status Updates happen only to people for whom the program is the last program with which they interacted
    • Debug improper workflows & update/maintain as needed
  • Reporting
    • Reports are properly built, as requested, in the program
    • Report subscriptions (either report or smart list) are only created for approved people
    • Subscriptions are deactivated when no longer needed


Region Admin -

Not every company will have this role and not every company will have a separate role for Region Admin / Marketing Approver, however there are functions that a Region Admin owns that are crucial towards success of marketing automation.  If your instance is differentiated by region you need to have a regional expert who natively understands the region and can act as an evangelist of that regions users. They also need to be part of the Marketing Operations Round Table, representing that region's needs and their knowledge of the users. The RA is the main point-of-contact for the region, providing training and Tier 1 support. This role does not have Admin access but has access to "secret" workspaces where critical-to-quality workflows are housed, out of reach of the Marketing Approvers. I'll cover more on this in one of my next posts about Big Marketo.


Your Region Admin has access to Lead Database and can change data directly when necessary. They have access to the Design Studio and can create and update templates for Email and Web Pages as well as Forms. (Look for an upcoming post about how Centralization Is Your Friend.) They can update scoring programs and the Lead Lifecycle.  The Region Admin is a powerful role and requires someone who works closely with Sales and Marketing leadership and understands business needs and how to continuously improve programs.



  • Maintenance, creation, activation and deletion of all region specific workflows
  • Attention to notification logs to proactively find issues and correct
  • Escalation to Admin for issues that are affecting or affected by other regions
  • Communication with other Region Admins for proposed workflows or programs that may be considered Best Practices
  • Coordination with other region admins to schedule large imports or complex batch programs that will slow down the campaign queue
  • Training and support for users within that region
  • Assessment and feedback to Admin of potential up or downgrading of Marketing roles
  • Make changes to templates and batch update assets
  • Add / maintain centralized forms and workflows
  • Level 1 troubleshooting for the region


Admins -

These are the highest, most proficient Marketo users. The requirement is that they be a Marketo Certified Expert to even be considered, be part of the core Marketing Operations team, have clearly defined knowledge of Markeo and share a unified vision for preserving the integrity of Marketo. This group should have an extremely conservative mindset when it comes to altering Marketo.  Requests for new fields need to be scrutinized for unanimous agreement that there's no alternative for storing data. They need to know how to write webhooks, how to partner with organizations looking to utilize APIs and how to troubleshoot integrations.


You should limit your Admins to as few people as possible, with an eye on spreading out responsibilities across timezones to provide around-the-clock support. For GE Healthcare, there are 4 admins: 2 in India, 1 in Australia and 1 in the U.S. We work closely together an act as one mind.  We listen to the needs of others and work to create solutions. We do not utilize our Admin powers often and we also act as Region Admins and Marketing Approvers.



  • Changes that will affect the entire Marketo instance:
    • Field additions or name changes
    • Program statuses
    • Tags
  • Opportunities to improve the Marketo instance performance and reliability (i.e. SPF / DKIM)
  • User Roles and User Accounts, including LaunchPoint
  • Proactively search logs for issues affecting performance / integrations and coordinate with RA for fixes
  • Involved in new API / Webhook connections
  • Work with designers / RA to test and approve templates
  • Create support tickets if issues cannot be resolved with internal resources
  • Activate new features and create training plan for RAs
  • Work on implementing new features / modules of Marketo and create documentation and training materials


Now, there are two other roles worth mentioning:

  • API User

    • A role for APIs wishing to integrate with Marketo. Mandatory for the creation of an API account
  • Agency

    • Hybrid roles created based on the reason and skillset of the Agency.  Typically an Agency will never have access to the Lead Database, Admin Functions or be able to approve or execute a program without vetting from the Admins for skill, understanding of business and security.


These are the roles we utilize at GE Healthcare. All of our 160+ users fit into one of these roles and can perform the duties that their job requires to the fullest.  This is also thanks to the program structures and centralization techniques we're utilizing to keep the instance sane.


Now it's also worth noting that these roles are only most crucial in a Production environment.  We often have Content Editors or Marketing Approvers who want to model changes or new programs which may require new fields. We require all work of this manner to be performed in a UAT / Sandbox environment and have reviews by the Region Admins and Admins prior to migration to Production.  This is where the Admins have the opportunity to train users on how to accommodate programs without the addition of new fields, or come to consensus on whether or not to add new fields as well as provide rigorous testing on proposed new programs or changes.  This is where Admins, who have knowledge of what other regions are doing, can be of most benefit by using on region's innovation to benefit others.


So ends Pat one on Big Marketo.  Let me know what you thought, how you'd improve on what I've detailed or what you'd like to see in a future installment.

In my previous posts on the "sync with Marketo" "Sync with Marketo" mysteries part 1 and "Sync with Marketo" mysteries part 2, I pointed out the fact it's impossible to know through filtering that a record has stopped sync'ing, except through visiting lead activity logs one by one.


The typical use case is:

A sales person learns a contact has left the company. the salesperson flags the contact as Inactive in SFDC. A workflow automatically unflag the "sync with Marketo" field, the sync immediately stops, meaning that Marketo does not event know the contact is in fact inactive.


In fact, I recently and inadvertently discovered an interesting behavior of the Connector which drives to the possibility to know...


It is based on the fact that SFDC formula fields cannot be updated by Marketo, but always sync from SFDC to Marketo even if the sync raises an error. The only case in which they do not sync from SFDC to Marketo is when the sync is stopped.


Create a formula field in lead and contact objects. Call it "Sync test" or anything else. Put any formula in it. If you want to make it a useful info, copy the SFDC record id into it, you may need it one day (I'll right another post on this).


When you want to test the sync on a record, Create the following smart campaign:


Smart List

  • Trigger: campaign is requested
  • Filter: SFDC Type [Person] IS NOT EMPTY (no need to do this on a record that has ever sync'ed)
  • + any other filter you see fit



  1. Change data value, attribute is "sync test", new value is NULL
  2. Sync lead to SFDC (assign to whatever you want)


Qualification rule:

  • Always run through the flow


Run the smart campaign on the leads you want to test.


When the campaign is over, if the sync is stopped, the "Sync Test" field will still be empty. If the sync is not stopped, the "Sync Test" field will be filled again with the formula value EVEN if the sync did not occur (for instance because of sync error, due to validation rules in SFDC).


The interesting thing is this is that the blocked sync records in Marketo can therefore be queried with the "Sync Test IS EMPTY" filter, for instance to delete them from Marketo.





End of Year for Marketing Automation

The end of year is around the corner! And it's the time of year to clean everything up. Prepare for the next season. And of course to support sales with their final deal. It wouldn't be the first time some new KPI's are introduced late January, but tracking wasn't enabled as of January first. If only you would have known in advance!

I have described 27 marketing automation end of year tips in 5 categories. I hope these provide you some additional ideas to consider for your own end of year checklist. Though I'm mainly focused on Marketo, most of the tips on the end-of-year checklist are platform independent.If you have any additions, please leave them in the comments below.

  • A. End of Year Marketing Campaigns
  • B. Marketing Performance
  • C. Marketing Operations
  • D. CRM & Marketing Automation Data Quality
  • E. Marketing Budget & Targets


A. End of Year Marketing Campaigns

1. Holiday Season Campaign

Keep the momentum going! Out-of-sight is out-of-mind. So make sure you've scheduled some engagement throughout the holidays. Suggestion: Try something with a gift. That Always works this time of year.


2. Draft next year's Campaign Calendar

Don't stop with the holiday campaign. Make sure you have at least your Q1 marketing calendar ready. You don't want to start the year with a blank calendar!


3. Turbo-charge that hockey stick!

Are you in the software space? Perhaps in high-tech or business services? There's a chance you encounter a hockey stick effect every year. Some companies make up to 50% of their profit in the last month of the year. If you haven't reached your year targets yet, you might want to consider teaming up with sales to support them in closing deals. Crank up your account based marketing programs for the top deals in the pipeline.


4. Archive campaigns and folders

Maybe this is something for January 1st, but you could consider it right before the holiday starts. Check out your campaigns and folder structures in tour marketing automation platform. Archive campaigns, which are no longer needed in the next year. It will increase performance, as this programs are not loaded in picklists anymore. And your navigation is nice and clean for your users when they start the new year. If you use 'year' in your folder names, you could already set up new folders for each business unit for the next year.


B. Marketing Performance

5. Align with leadership

People change and so do their ideas. Don't assume leadership will have the same priorities for the next year. Meet up and discuss if any of the KPIs currently used need to be replaced.


6. What worked?

The end of the year is great time to evaluate what worked and wat didn't. So many people do this after the year has ended, whilst they already created the new marketing calendar. You want to make sure to evaluate what worked and what not, BEFORE you create next year's marketing calendar.


7. Check in with (sales) colleagues for tips for next year

They sales is busy closing the year, reach out to your colleagues and ask them for input. How did they experience this year? Do they have any tips for next year?


8. Prepare your end of year reports

Steal the show be being the first to report on 2016's performance!


9. Share best-practices

Even before everyone starts their next year's marketing calendar, organize a best-practice sharing session. You which campaigns worked or not. Invite those owners to share their pitfalls and learnings with the rest of the marketers.


10. Check your report settings

Quickly check how your reports are set up. Some reports use a fixed year (e.g. 2016). So they won't work in the new year. So these need to be cloned and updated with the correct year. Or you make the year dynamic (e.g. 'this year').

C. Marketing Operations

11. Give kudos to your team

Here's an open door! Thank your team for all their efforts to make marketing automation succeed at your company.


12. Compare the marketing operations roadmap

How did your marketing automation roadmap look like at the start of the year? What was accomplished? And what not? What does the roadmap look like for next year? How do they compare? Align met the sales operations team and/or CRM team? How do you align? How can you align better?


13. Set up those new business units

New year new business structures. Some companies choose to allows start new business units on January 1st, to have proper reporting. Make sure everything is set to go on January 1st though. Think about field picklists, account ownership, user access, campaign templates, reporting, etc.


14. Daily hockey stick updates

Do you experience the hockey stick effect, mentioned in #3? Update your commercial colleagues on a daily basis on the hockey stick progress. Drive that revenu!


15. Contract renewals

The hockey stick effect is also a reality for your suppliers. In marketing automation's case it involves your entire marketing technology stack. The end of year is good for getting more discount (if applicable). Though negotiations are harder at the end of year for bigger platforms. They well understand you won't pull the plug in the last few days of the year. As you will start the year without any platform to do marketing. So for bigger platforms, start negotiations in October! if you don't like where the negotiations are going, you will have plenty of time to switch before the end of year.


16. Educate (the team)!

You don't get all fuzzy inside when the holiday season is nearly there? If you are not planning to attend mandatory drinks or family dinners, you will now have plenty of time to stay up-to-date. Educate yourself on what's happing in the marketing landscape. Get your team involved too!


17. Deactivate old users

I hope you do this on a regular basis. Or even better, you have a process for this. But check your marketing automation users. Who no longer works at your company? Who never logs in? Do they really need that many permissions? Keep the list clean, keep your instance secure!


D. CRM & Marketing Automation Data Quality

18. Check for synchronization errors

You don't Always notice these. Check random log files. Compare and export from your marketing automation platform and your CRM.


19. Monitor data drop outs

Like #18, check whether there are records, which are not linked to any business unit. Or simply check which leads have fallen through the cracks. Make sure you end the year with good data. You don't want bad data to pile up every year.


20. Complete competitor lists

Are there any new competitors? Check your data workflows and update them where needed.


21. Check accounts and target lists

New year, new targets for sales. And sometimes target account lists are redistributed. Or business units shift their industry focus.


22. Archive old leads

Are those yearly list imports growing your database with garbage? Most marketing automation license fees are based on the amount of leads you have in the platform. Export leads without any activity for over X years. Then remove them. Keep your database healthy. End of year is the perfect time.

E. Marketing Budget & Targets

23. Budget versus cash flow

You might know your budget, but that doesn't mean you will be able to spend it completely in January. Check with a controller from the finance department what your options are. We Always want to beat the end of year marketing budget cuts (to lift profit), by spending it all in the first months of the year ;-)...... Well I hope you work at a company that doesn't see marketing as a cost center though ;-)


24. 'Waste' your remaining budget for this year

Thinking about #23, you might want to consider checking how much budget you haven't spent yet. You don't want your manager to think you can achieve the same results with less budget. That would automatically result in a budget cut for next year. I assume that, as a marketing automation user, you are very successful. And you want to innovate and improve every year. Driving company growth! You need more budget! Not less! So make sure to spend your budget!


25. Update conversion rates

A lot of your calculations (e.g. targets) depend on the conversion rates you achieve. How did those rates evolve this year? Should you make adjustments?


26. Calculate next year's targets

Next to budget, you also need to calculate marketing's targets for next year (together with leadership of course). Make sure you've got support on the new targets before the end of year. And make sure you can start reporting on progress on January 1st.


Do you have any additional suggestions?

27. Share your own best-practices with others

If you have any suggestions for additional items, or if you have any other feedback, please leave a reply below. It would be the perfect holiday gift for your Community peers ;-)


* This article also appeared on

Hey #MKTGNation!


On this week's episode of #KreweChats, we delved into Marketo Sales Insight with special guest (and #MKTOChamp) Emily Dick. Emily is an absolute UNICORN when it comes to Marketo and Salesforce expertise, and we were thrilled to have her perspective on this episode of #KreweChats! More professionally speaking, Emily is a MOPs Manager at ACL, Marketo Champion, and an all around great person to chat with about all things Marketing. Also seen on this chat were #MyKrewe regulars Jenn DiMaria, Sydney Mulligan, and Juli James!


As many of you are probably already aware, Marketo Sales Insight is an amazing tool that can help any organization drive towards better alignment between sales and marketing, and it does this in a few unique ways. You can dive into the full episode at the link below, but, um... This is a blog post, so we gon' expound on some shenanigans.


  • Point #1: you learn more about the Krewe's favorite fast food indulgences than you ever wanted to know. Shout out to my Louisiana folk
  • If you aren't putting MSI into a frame of reference that immediately benefits your sales team, you're doing it wrong.
  • You should also be conducting regular training for your sales team. We recommend a monthly webinar series that walks through each aspect of MSI (or quarterly, at the very least!)
  • Make it easy for your sales team to send emails through MSI, whether directly from SFDC or the Outlook Plugin (Sorry Mac users/AKA: Listen up, MKTO!)
  • Encourage your sales folks to prospect via the Anonymous web activity tab and LinkedIn Integration!


Those are the high points, but if aligning your sales and marketing organizations is something you're interested in, then this episode is for you!


*#KreweChats Episode 9: Marketo Sales Insight - YouTube

I actually started my career at a BI company that was doing unbelievable Marketing reporting like 10 years ago. At the time, I completely took it for granted, and now after working with hundreds of companies, realize how far the Cognos team was ahead of the curve.


Fast forward to today, and companies are pushing the reporting boundaries way further than I ever though possible. Two great examples of that are Informatica and Trend Micro.


Over the year's I have got to work with some of the most talented Marketo practitioners, but when it comes to reporting, there is no one better than Anish Jariwala. Anish used to be a principal consultant at Marketo, doing reporting and analytics for some of Marketo's largest customers. He is the master of RCA, and knows it inside out. Anish recently wrote a book on Marketing reporting, with a big focus on Marketo reporting, called 'The Marketing Data Lake'. The book is essentially a blueprint on how to become a world-class marketing reporting organization. I highly recommend the read. I will be honest, reading a book about reporting and analytics was not at the top of my reading list, but its a surprisingly light, easy read.


One of our client's, Trend Micro, read Anish's book and applied many of Anish's best practices and approaches to their own Marketing. They were able to go beyond Marketo Revenue Cycle Analytics and integrate Marketo with Tableau. The reports they are now able to produce allow them to guide their Marketing decisions like never before, and the numbers are being reported up to the highest level in the organization. Brendan Farnand and the team at Trend Micro are now making way more informed Marketing decisions, that were simply not possible before.


So - I was thinking. Anish and Brendan have so much knowledge that the Marketo community could benefit from. I want them to share this with everyone so people can see what's possible when it comes to Marketing reporting, and how to actually do it.


We're doing a Livestream using Crowdcast on Wednesday, November 16 from 1-2pm ET / 10-11am PT. If you want to learn practical tips and best practices to improve your Marketo reporting, there is no better session. Hear from the people who have actually done it, and have the scars to prove it. Ask questions, upvote other questions you might have, and leave with a better idea of how to get those reports you've been wanting. This will not be a session with a million PowerPoint slides, it will be live and interactive and you will have a chance to participate.


We hope you'll join us. Register here: Marketo Reporting 2.0


Social Ad - v1c.png

Joe Reitz

Marketo-Fu: Vlog Launch

Posted by Joe Reitz Champion Oct 27, 2016

Hey #MKTGNation!


A couple weeks ago, I started recording a daily (ish) live YouTube series on "how to Marketo." That's a dumb name, so I opted instead to call it Marketo-Fu (literally made up on the fly, like 30 seconds into the first video). This is a very down-and-dirty, genuine series, where I decided to dump the polish of some of the other Nation Talks videos because... well, honestly, these are easier and quicker to make, and still just as valuable in terms of knowledge development! I try to make these short, single-serve topics at around 5-10 minute in length, so perfect for that 4:45 crunch where you don't feel like starting something new, but you also don't want to be the first person to go home, either


Don't even act like that's not a thing in your office. I know. I've been there


The ultimate vision for this series is to have a few different tracks for beginner, intermediate, and advanced users, and continually add episodes to the playlists as either:


     A) I think of them

     B) my clients would find them useful reference material, or

     C) You ask for them.


I just completed the 10th episode in the beginner track and will be moving on to the more Intermediate and Advanced tracks in the weeks ahead, but I wanted to post here to share and get the conversation going!


You can watch the first 10 episodes here, and start making your Marketo-Fu strong! Marketo-Fu (Beginner) - YouTube!


So... What other stuff would you like to see covered on a future episode? The tentative line up for the Intermediate/Advanced Tracks includes:


  • Lead Scoring
  • Alerts
  • MSI Overview - Interesting Moments, Prioritization, etc.
  • Bounce Management
  • Merging Dups
  • Advanced Flow Steps – Request Campaigns, Remove from Flow, Change Data Value
  • A/B Testing - 3 methods (Spoiler: This exists already )
  • Attribution – First Touch & Multi Touch
  • Web Personalization
  • ABM




Hey Marketo! We're at it again with another two episodes of Krewe Chats! Two weeks ago, and also today we covered Lead Lifecycle and the Revenue Cycle Modeler. It was way too much to fit into just one episode, so we had to cut our first chat short and follow up on the next episode.


In these episodes, we cover a ton of useful lifecycle stuff! MQL, SQL, OMG!! (I just like saying that, but hey... who doesn't?!) Juli James, Sydney Mulligan, Jenn DiMaria, Rachel Egan, Geoff Krajeski, Joe Reitz and I get down into the nitty gritty of first steps to setting up the lifecycle modeler, who should be involved in the conversations, and then how you can actually get the ball moving, and what to do with the data once you're done.


Check out Episode 7 here: #KreweChats Episode 7: Lead Lifecyle & Revenue Modeler - YouTube

And Episode 8 here: #KreweChats Episode 8: The Revenue Cycle Modeler - YouTube


In these episodes we cover:

  • Our favorite random facts, and favorite beverages
  • Common stages that you might see in a lifecycle model
  • Who else should be involved in the conversations about setting up a lead lifecycle? (Hint: Not just marketing!!)
  • What are some of the key questions that you should be asking in these initial discovery conversations?
  • How do lifecycle stages translate and tie into Marketo's Revenue Cycle Modeler?
  • Now what in the world do we do with all of this data?!


Also, here's the lifecycle that we shared during the chat. This is basic, but can always be tweaked to your specific needs!



We look forward to our next chat, be sure to join us on November 4th, at 3:30pm ET for the next #KreweChats



Check out Episode 7 here: #KreweChats Episode 7: Lead Lifecyle & Revenue Modeler - YouTube

And Episode 8 here: #KreweChats Episode 8: The Revenue Cycle Modeler - YouTube

We’ve all used Marketo or other automation tools to A/B test emails and landing pages. We do it because we want to optimize engagement through constant iterations, and we can use the results to give our content its best shot at provoking responses from our prospects.


But have you ever had the nagging feeling that your high school statistics teacher wouldn’t approve of your testing technique? You remember terms like sample size, variables, and p-value that were important parts of your hypothesis testing, but they all seem to be missing in Marketo’s tool today.


It turns out those principles we learned are still integral to executing a successful A/B test and preventing incorrect conclusions. Luckily you don’t need a stats degree to implement these principles and enhance the tests that your organization performs. So let’s dive into how to design and interpret a more meaningful Marketo A/B test.


I.  Designing your A/B Test


Your test design is the most important factor in determining whether you will get insightful information from your results. Over and over, we see the same common experimental design fallacies in tests run by marketers. Let’s take a look at what they are and how to overcome them.


Sample Size is Too Small


How large does my sample size really need to be? We get this question a lot and wish there was a definitive answer. But we would by lying to you if we said there was because it depends on how big the difference is that you want to see.


Say you want to do a simple subject line A/B test and you send to 1000 recipients.


Subject Line A: [Webinar] How to make the most of your A/B tests

Subject Line B: [Webinar] Register Now: How to make the most of your A/B tests


Half get Subject A and half get Subject B. If 6% open A and 7.4% open B, can you draw the conclusion that having a CTA “Register Now” performed better? Is the difference between A and B significant enough to declare that B is “better”? We can’t really answer that until we look at the p-value and how to get the p-value, which is covered later. For now, smaller p-values are better and in this case the p-value = 0.376 which is not good. You might think “Subject Line B still got higher number of opens, so why don’t we just go with that?” What the results are also saying is that the chances of you getting the opposite results if you ran the test again is pretty high.


If we run the test with 10,000 recipients total with the same percentages opening A and B respectively, the p-value is significantly smaller at 0.0051 which is excellent. (Scientific publication guidelines accept <0.05 and this is just marketing.) With the results from the second scenario you can confidently conclude that adding a CTA makes a difference. The combination of your target size and the difference between your two test groups determines what conclusions you can draw from your results.


Changing too many variables at once


As marketers we get excited about testing different variables. Sometimes we go overboard and test too many variables at once which leads to the failure to conclude anything. Let’s demonstrate with a landing page test.


Landing Page A: Blue button with CTA = Submit

Landing Page B: Green button with CTA = Download Now


In this case we have a question: Which button performs better? If Landing Page A has a significantly higher conversion rate than Landing Page B, what is my actionable intelligence moving forward? Unfortunately, we do not know if it is the color or the words on the button or both that was the contributing factor. (If you want to geek out this is called a confounded experiment.)


The proper way to carry this out is to break out the testing out into two rounds.


Test #1

Landing Page A: Blue button with CTA = Submit

Landing Page B: Green button with CTA = Submit


Result: Landing Page A performed significantly better.


Test #2

Landing Page A: Blue button with CTA = Download Now

Landing Page B: Green Button with CTA = Download Now


Result: Landing Page A performed significantly better.


Conclusion: LP with a blue button and an active CTA should be implemented.


If you vary multiple factors at once in the two test groups, you will not be able to conclude which of the variables that you changed contributed to the performance of one group over the other. Setting a series of tests to vary one variable at a time allows you to truly understand the contribution of each.

Testing without a clear question or hypothesis


Have you ever carried out an A/B test and then asked yourself “What do I do with the result? How can I apply this to future campaigns?” This confusion often occurs because you designed your test without a clear hypothesis.


Here’s an example of a subject line test with 6 groups.


A: Learn from CMOs: Engagement Strategies

B: How to effectively market to your prospects

C: Top strategies for engaging your prospects

D: Top strategies for reaching your prospects

E: Web Personalization: Reach and engage your prospects

F: Drive greater engagement this holiday season


If subject line C was declared the winner with the greatest number of clicks (albeit by a slim margin), what have we learned to apply for the next time? Also, with this many variables you will need a very large sample size to declare this result to be significant.


A better strategy would be to break out into a series of tests where we can test a single variable at a time with a clearly defined question or hypothesis.


Question #1: Does having CMO in the subject line drive more opens?


Subj A: Learn from CMOs: Engagement Strategies for your Marketing

Subj B: Learn Engagement Strategies for your Marketing


Question #2 Does the word “reaching” or “engaging” drive more opens?

Test #2 (Assuming CMO won test #1):

Subj A: Learn from CMOs: Top strategies for reaching your prospects

Subj B: Learn from CMOs: Top strategies for engaging your prospects


Question #3: Does mentioning “holiday season” results in a greater open rate?

Test #3 (Assuming reaching won test #2):

Subj A: Learn from CMOs: Top strategies for reaching your prospects

Subj B: Learn from CMOs: Top strategies for reaching your prospects this holiday season


Remember that it’s called an A/B test, not an A/B/C/D/E/F test. Break down your question into specific parts that can be tested in a series of A/B tests, rather than trying to get an immediate answer by testing all at once. The next time you are deciding what individual elements of a subject line will maximize engagement, you can look back at the results of these tests.


Using the Email Program A/B test results to declare a “Winner”


In the Marketo, it is really easy to set up an A/B test using the Email Program and see the results. Let’s go back to our simple subject line test for registering for a webinar.


Subject Line A: [Webinar] How to make the most of your A/B tests

Subject Line B: [Webinar] Register Now: How to make the most of your A/B tests


Say you have 50,000 leads in your target list and you choose to test 20% of your list and send the remainder the winner. That means 5,000 will get subject line A and 5,000 will get subject line B. The subject line that is declared the winner will be sent to the remaining 40,000. That sounds pretty straight forward. But (and you knew there was a but...) how is a winner determined and which one should you choose?


Marketo lets you set the winning criteria and automatically send the winner a minimum of 4 hours later. You can choose from the following:




Clicks to Open %

Engagement Score

Custom Conversion


In this case if we choose opens, that means that the difference in the subject line is the difference in whether someone opened the email or not. Is this the behavior that matters most? In some cases that might be, but in a webinar we probably want to look at clicks instead. For example, we once saw an email that had the larger open rate also had less registrations and a 10 times higher unsubscribe rate. This led us to conclude that our message was not resonating with the target audience.


Setting the winning criteria to Clicks to Open % could also be problematic. If email A had 1000 opens and 40 clicks (4%) but email B had 200 opens and 20 clicks (10%), email B would be declared the winner even though the absolute number of people who clicked is lower.


What about setting the winning criteria to clicks? If Email A had 1000 clicks and Email B had 100 clicks, Email A would be declared the winner. But if the desired behavior is registering for the webinar and Email A had 10 people register for the webinar vs 25 for Email B, was email A really the “winner”?


So… which one should you pick?


Unfortunately you won’t know until you look at the data after the results come in. There is no way to predict. We can think of a potential situation where any of the choices above would work or not work, it will just depend on what the data says. So if you are going to declare a winner n a Marketo A/B test, we prefer to do it manually.


“When I test, I typically test on 100% of my target list. If I have an A/B test with 2 groups, I set the slider bar to 100%. That way, 50% get A and 50% get B. I do this for a number of reasons. Because, you won’t know if you have a large enough sample size until after the test. If you run 10 different tests on 1000 people and the difference is small, your results will all be inconclusive. I would rather run 1 test on 10,000 targets and get a really solid conclusion.“


When you are designing a test, ask yourself, “What am I going to do with this information? What am I going to change?” Don’t test for the sake of testing. Whatever you decide to test, ensure that the question you are asking is going to be actionable. Now that you know how to design robust A/B tests, how do you interpret those results?

II.  Testing and Interpretation of Results


  Setting up the test correctly is half the story, making sure that we are drawing the correct conclusions is the other half and just as important. 

Unfortunately, we cannot “declare a winner” by simply picking the test group with the most opens or clicks.  When we run a test we are saying, this small population of 1000 people is a representation of the whole universe.  It is not possible to test everyone in the whole world.  We are extrapolating that how this sample population behaves is going to predict how the rest of the world would behave.  But. . . we know that if we ran the test on 10 different sets of 1000 people, I would get slightly different results, so there is a chance albeit small, that I might have picked a sample population that is an outlier so different then the rest of the world my results could lead me astray.  This slight variation is what we need to account for by calculating a p-value. 


Let’s go back to our subject line test.


If you sent a total of 1000 emails and 30 people opened email A and 31 people opened email B, could you say email B leads to more opens? The answer is no (based on the calculation of the p-value).  Just because Opens of email B is > than opens of email A doesn’t mean that if you hypothetically ran the test again you would get the same results. In this case it’s about as good as flipping a coin. You could get either result. 


The real question in A/B testing is:  “Is the difference between A and B SIGNIFICANTLY different enough for you to draw the conclusion CONFIDENTLY that B is greater than A when you run the test again and again.  You want to be able to confidently say, based on the results of the test, I believe B will most likely yield more than A if I were to run the same test in the future.  Therefore, we should move forward with B.  That’s the goal.


To determine whether the difference is significant or not we look at the p-value of our test.  We are not going to go into how this value is calculated, but we will examine:

  1. How to use a very simple tool to obtain the p-value
  2. How to interpret the p-value
  3. What it means in plain english


You can use this website to input the results of your A/B test and generate a p-value.  (This calculator was posted by @ Phillip Wild.  A/B Testing and Statistical Significance.  Great suggestion) 

Let’s take a look at another example. 

You run an Email A/B test separated into groups with two different button colors, green and blue for the call to action. Your question is which button color is associated with more clicks. 

Green: 93 clicks on 4,000 emails delivered

Blue: 68 clicks on 4,000 emails delivered


You take the number of clicks for each group and plug them into the Calculator test under the successes for each group. You enter 4000 into the total for each group. 


The resulting two-tail p-value = .047. 

It is generally accepted that a p-value of <=0.05 is considered a significant result.    The smaller the p-value, the better and more confident you can be in your results.  We can conclude that there is significantly higher number of clicks using Green vs Blue.  I am confident that if I were to run this experiment again and again, I would obtain the same result.  Therefore, I would make the recommendation to change the CTA button color to green. 

What does this p-value number mean in plain english?

A p-value of 0.047 is saying is that there is a 4.7% chance that you could have obtained these results by random chance and that if you were to run this experiment again you would not see the same result. 


What is so special about a p-value cut off of 0.05?

It is in fact an arbitrary cut off but is the absolute gold standard and is used in the scientific and medical community in the most highly respected peer reviewed publication.

If your p-value is slightly more than .05, say .052, don’t automatically write off the result as inconclusive. If you have the ability, test the same hypothesis again with a different or larger sample size. 


Note:  When using this tool, plug in your number of successes (opens, clicks etc.) and total (number of delivered emails) for each group. Note that when using click to open ratio, you will be using number of clicks as the success and number of opens as the total, NOT the number of emails sent.


This calculator gives us the p-value of the test, and we want to look at the two-tail value specifically. The p-value of a two-tail test represents the likelihood that there is a statistically significant difference in what we are measuring between the two groups in the test, compared to when there is actually no true difference. If the p-value is smaller than .05 we can conclude that there is a 95% or more chance that there is a difference between the two statistics (open rate, clicks) and act upon that in our decisions for future marketing communications. If the p-value is above .05, then the results of the test are inconclusive. This value and interpretation allows us to stay consistent from test to test.


A key here is to not consider the test a failure if the results are inconclusive (p-value is greater than .05). Knowing that changes to certain email content or timing won’t likely have an affect on your audience is just as useful for future communication strategies. If you still feel strongly that the first experiment wasn’t enough to capture the difference in your groups’ responses, then replicate the experiment to add to the strength of your results.


Organizing your results for future use


“As a lab scientist, I was taught to keep meticulous records of every experiment that I did. My professor once said to me, if you got hit by a bus or abducted by aliens I need to be able to reproduce and interpret what you did. As a marketer you probably don’t need to be that detailed but nonetheless it’s nice to have a record of what you have done so you can refer back to but more importantly share with your colleagues. For testing marketing campaigns, I kept a google doc, excel sheet, or a collection of paper napkins (true story). “


Keep a record of what the test was, the results, and the conclusions. And don’t be afraid to share your results in a presentation once a quarter. You immediately increase the value of your hard work by sharing your findings with your organization.


Here’s an example of a test result entry:


Aug 4, 2015

Test day of the week


Target Audience: All leads with job title = Manager, Director, VP

10,000 Leads


Email A - Send on Wednesday 10 AM

# Sent = 5,000

# Opens = 624

# Clicks = 65

# of Unsubscribes = 68


Email B - Send on Sunday 10 AM

# Sent = 5,000

# Opens = 580

# Clicks = 94

# of Unsubscribes = 74


P-value (Opens) = 0.176

P-value (Clicks) = .020

P-value (# Unsubscribes) = .612


Conclusion: Emails sent on Sunday resulted in more clicks, but there was not a difference in opens or unsubscribes.


If you clearly document and organize your test results, you’ll soon have a customer engagement reference guide that’s unique to your organization.  And if you’ve designed your experiments as advised above, you’ll know that the conclusions drawn are based on sound statistical analyses of your data. Put those “fire and forget” Marketo A/B tests to rest and you’ll make your way towards optimal customer engagement.

What is your experience with Marketo’s A/B testing? Have you found any results that are interesting or unexpected?  Feel free to share your experiences with testing.


I'd like to thank Nate Hall for co-authoring and editing this blog post. 

See the video here: #KreweChats Episode 6: The One About ABM - YouTube


Learn each of our favorite songs!! (Playlist available here: KreweChats Favorite Songs - YouTube)


Watch our panel of Ande Kempf, Dory Viscogliosi, Juli James, Sydney Mulligan (aka SMUGS), Jenn DiMaria, Joe Reitz, and guest chatter Brent Evans discuss the ever popular topic of ABM, Account Based Marketing.



Account Based Marketing or ABM is the latest and greatest buzz word in the marketing world.  In this episode we discuss what it means to us as Marketo has just announce the release of their new ABM module that can be added on to our subscriptions.


Many of us have or will soon be getting our sneak peek at this exciting development, but wanted to bring our take to you and how this will be the path forward toward unifying Marketing and Sales teams for companies in achievement of further returns.