Predictive Testing of Opportunities – Frequently Asked Questions

What the PTO is. How it works. What you get.

  1. What is PTO? 
  2. How is success defined?
  3. What does PTO tell you about why an opportunity will succeed or fail?
  4. How does PTO work?
  5. Tell us more about the analytics simulation
  6. How can a general model consider the attributes and behaviors of a specific company? Wouldn’t a model need to learn the company’s strengths and weaknesses (competencies) to be truly effective?
  7. How can you calculate all of this if you don’t have detailed data on our industry, our markets, our technology and our company?
  8. What data do you harvest? What are the sources?
  9. Does your model use data from companies like us (i.e. in our industry)? For what industries does your model work ‘best’?
  10. What is the accuracy of the prediction?
  11. How do I know if the results are accurate? How has this been tested?
  12. Does this accuracy hold across industries and technologies?
  13. Do we get to see which opportunities make up the dataset of 4000?
  14. Is there a difference between B2B vs. B2C predictions? Between products or services?
  15. How granular is the analysis? Can you analyze a specific technology within a given industry?
  16. Can the PTO analysis predict yearly sales volume?
  17. We use BASES to predict the success of new products before they are launched. How does this compare?
  18. Can PTO be used for operational innovations (i.e. innovations in the way we manufacture etc., not in new products or services)?
  19. In the Clorox Greenworks example, growth is predicted at 2%, what has it been?

How do we use the PTO service?

  1. How do we choose which opportunities to run the simulation on?
  2. How do we collect the information to fill out the input forms?
  3. What if we don’t have enough information? We don’t just like to guess at things that are so important
  4. How do we get started?
  5. What information do I need to provide?
  6. What happens to the information we provide?
  7. What if the inputs are wrong or if they are biased by the person providing the information?
  8. What if I don’t have the necessary Information?
  9. What if the opportunity is at a really early stage and is only a vague concept?
  10. Can I get a copy of the filled in input form?

PTO Results and Deliverables

  1. Does it work for global opportunities (offerings sold or developed in other regions)?
  2. Can this be used to look at M&A opportunities?
  3. What is in the report that we get for each opportunity?
  4. Are the factors that contribute to an opportunity’s success or failure broken out?
  5. What is the workshop? What do we do there?

Business Information

  1. What PTO services does Inovo offer?
  2. What are the advantages of working with Inovo?

What the PTO is. How it works. What you get.

What is PTO?

PTO stands for ‘Predictive Testing of Opportunities’ and that is exactly what it does. It is a service that predicts whether an opportunity you are now developing (or thinking about developing) will be successful. In addition, it estimates the expected growth rate of an opportunity.

The opportunity you choose to predictively test can be anything from an early stage concept to something that is about to launch. It can be a product, service, or new business model. It can be B2B or B2C. It can be for any industry or market and can use any technology. It doesn’t matter. PTO can predict if the opportunity will be successful with approximately 67% accuracy – over twice the accuracy that research has shown companies achieve today.

How is success defined?

There are many measures of ‘success’ and we try to stay out of debates around what “success” is or isn’t. That said, we predict 2 separate things.

  • Likely mortalitywithin 5 – 10 years (will the product line/business model exist or be terminated in that timeframe)
  • Likely annual revenue growth ratewithin 5 – 10 years.

Success, as we define it, means that the product/service/business is still in existence in 5 – 10 years. This is used in the modeling because it is the most objective proxy for “success”, and is not open to interpretation.

Note that in benchmarking performance against alternative prediction approaches both are held to the same standard: predicting if “the business survived”. By that metric, the PTO approach is much more accurate.

What does PTO tell you about why an opportunity will succeed or fail?

The PTO not only predicts the success and growth rate of an opportunity; it provides insight into the causes of success or failure. These insights are of four distinct types:

  1. Competitiveness – how much competition the opportunity will face and its competitive position. Competition includes both direct and indirect competition.
  2. Parent Company Fit – how well the opportunity ‘fits’ the company and how tightly is it integrated with the company.
  3. Market Momentum – how fast the market is changing, growing – includes technological factors as well as market demand factors.
  4. Market Profitability – What is the general profitability of the market the opportunity is addressing and how will the profitability evolve.

How does PTO work?

The PTO service consists of four steps as shown in the following diagram:

  1. Selection of the opportunity and gathering of data – this is done in collaboration with the company using assessment tools that Inovo provides (e.g. Opportunity Readiness, Should-Could)
  2. Running of the Analytics Simulation – Creating the input and running the simulation using the Growth Science analytics simulation
  3. Analysis and additional research – Creating a 10 – 12 slide report for each opportunity based on the results of the simulation and independent research done by Inovo.
  4. Working session to discuss and shape the opportunity – A physical or virtual workshop to go over the PTO assessment and shape the opportunity so that (if possible) it turns from failure to success and has a higher growth potential.

Inovo works with the client throughout this approximately 6-week process.

Tell us more about the analytics simulation.

The simulation takes place once an opportunity has been identified and described. The simulation uses your input data as a sort of “kernel” from which thousands or even millions of additional data points, using a variety of digital sources, are harvested. The simulation does this using proprietary algorithms, techniques and software. In other words, the vast majority of data used for each situation is not contained in your inputs themselves, but comes from the data mining and harvesting done later.

There are five steps in the simulation process:

  1. Opportunity input – Taking the opportunity description materials (documents and interviews) of the company and filling in the simulation input form
  2. Algorithm orientation – Taking the information in the input form and setting up the simulation parameters and data
  3. Harvesting relevant data – Searching for any and all data relevant to the opportunity as described using natural language process, thousands of potential data sources and intelligent searching algorithms.
  4. Modeling all possible outcomes – run the simulation thousands of times using randomly varied parameters based on the opportunity input and harvested data (Monte Carlo technique).
  5. Selecting the most likely outcome – Examine the probability distribution of all the outcomes and determine the most likely – success and growth rate.

The result is an assessment of the opportunity along the four dimensions of Competitiveness, Parent Company Fit, Market Momentum and Market Profitability along with supporting information derived from the underlying model and the harvested data. Inovo takes all this raw information and turns it into a 10 – 12 slide detailed report on the opportunity.

How can a general model consider the attributes and behaviors of a specific company? Wouldn’t a model need to learn the company’s strengths and weaknesses (competencies) to be truly effective?

The business analytics simulation uses three sources of information and knowledge to determine the likely outcome of an opportunity.

  1. An underlying, complex system model, built up over the past 10 years, using thousands of actual examples from which the significant, underlying patterns have been extracted.
  2. The harvesting of vast volumes of current data and information from thousands of information sources.
  3. Your description of the opportunity and your company. In describing an opportunity there is a lot of attention paid to the “Parent Company” within which the opportunity is being developed (and within which it must be successful).

We don’t just rely on what you tell us about your opportunity or your company, the simulation also harvests a lot of data about your company.  We pay attention to the peculiar behaviors of your company, both from what you tell us and what is available from external sources. We use this information to probabilistically compare with known patterns that have been captured by the underlying model. In this way, each analysis is company-specific – it considers the specifics of your company, and it can uncover the hidden patterns that make your company like (or unlike) others.

How can you calculate all of this if you don’t have detailed data on our industry, our markets, our technology and our company?

We ask for your honest assessment and opinion about your company, market, industry and technology.  However, when we run our analyses we use the data we harvest – not what you told us.  In other words, we want to know what you think, but we neither confine our analysis to it, nor do we necessarily believe you.  We rely 100% on the data we harvest vs. what you tell us.  For example, you may say that there are “no competitors” or that the business is “disruptive.” However, we use our harvested data and analytics to see if that’s the case.

What data do you harvest? What are the sources?

We literally use over 1000 information sources. Virtually any source that has an API we can access is included in our harvesting. Many of these sources are open and free, but many are behind ‘walled gardens (e.g. Facebook)) and/or charge for access and we pay to get access to them (this is one of the major expenses of maintaining the service). These sources go far beyond what you can get using Google (or any other search engine) although we use that information as well.

Does your model use data from companies like us (i.e. in our industry)? For what industries does your model work ‘best’?

Surprisingly, the determination of opportunity success does not rely on comparison with like offerings, companies or industries. We have found, from detailed analysis of the simulation and its results, that certain patterns of organization, technology, market and competition determine success and that these patterns cross all industries, companies, and offerings.

It is these patterns that have been learned and refined over time and which are constantly being updated and enhanced. These patterns are embedded in the underlying models the simulation uses. One can almost compare these patterns to that elusive term ‘business model’ more than anything else. It matters more to compare your opportunity with one in a different industry but with the same underlying patterns than it does to compare it with one in the same industry but with a different pattern.

What is the accuracy of the prediction?

The following chart shows the predictive accuracy of the PTO service.

Interpreting this chart can be a little tricky so here are some guidelines

  • Opportunities predicted to fail have actually failed 86% of the time. This means there is a 14% false negative rate (predicted it would fail and it actually succeeded). These represent lost opportunities.
  • Opportunities predicted to survive have actually survived 67% of the time. This means there is a 33% false positive rate (predicted it would succeed and it actually failed). These represent wasted efforts (i.e. opportunity costs).
  • In the aggregate, failure has been predicted more often than success (about 3 times as often).

The most interesting category for most companies is bullet 2 above, prediction of success. While the PTO has a 67% accuracy of prediction, humans (and the tools they currently use) have traditionally achieved only a 30% accuracy of prediction (at most). This 37% difference in predictive accuracy represents a lot of money and resources that could be better spent by the company.

How do I know if the results are accurate? How has this been tested?

The system has been validated with over 4,000 historic (real) opportunities that were not part of the training set. These are independent, real-world results of both success and failure.

Of course, ultimate validation of a specific opportunity requires the passage of time. That’s what occurred to get the 67% accuracy metric for the system. The 4,000 examples the model is built from are all ‘post-mortem’ (i.e. they are the actual success or failure of the opportunity). The accuracy results are not based on back-testing.

The outcome data set is continually growing as predictions that were made get added either when they fail (could be within the 10-year window) or when they succeed (i.e. pass the 10 year anniversary).

Does this accuracy hold across industries and technologies?

Yes. It does not matter which industry, technology or market the opportunity is for, the accuracy of the prediction is the same.

Do we get to see which opportunities make up the dataset of 4000?

While the makeup of the database is proprietary, we do provide an ‘analog’ example as one of the outputs of a PTO analysis. In addition, there are several cases of actual opportunities assessed by specific companies.

Is there a difference between B2B vs. B2C predictions? Between products or services?

PTO works for either B2B or B2C and for product and/or service opportunities equally well. The nature of the opportunity (B2B or B2C, product or service) is captured in the opportunity description and is used in the data harvesting, simulation and analysis.

How granular is the analysis? Can you analyze a specific technology within a given industry?

The focus of the PTO assessment is on the opportunity. An opportunity is a specific offering and business model that addresses certain customer needs with solutions that use certain technologies and are sold into competitive markets. In this sense, the description is very granular, you need to describe the offering you are contemplating.

This is not to say that every specific detail of the opportunity needs to be worked out. The PTO service is perfectly suited to opportunities that are at the very early stages of conceptualization and the results can often guide the development of the opportunity along the right path.

Can the PTO analysis predict yearly sales volume?

No, the PTO only predicts the potential growth rate for the opportunity over a 7- 10 year investment horizon. Specific sales volumes and revenue depend on so many executional factors (i.e. the effectiveness of your sales and marketing campaign, product quality, service and support, supply chain factors etc.) that PTO does not consider. The growth rate the PTO comes up with is your potential, whether you reach this potential or not is up to how well you execute.

We use BASES to predict the success of new products before they are launched. How does this compare?

BASES is a tool from Nielsen used, primarily by CPG companies, for near term unit/sales forecasts for mature product lines and markets.  It is an excellent tool in those situations but is woefully inadequate for innovations that are farther from the core business.

PTO does long-term, 10yr predictions for comprehensive business models (not just product lines within those business models) and for new offerings that are farther from the core (1.e. far adjacent and transformational opportunities).

Companies should use BASES for nearer-term core product forecasts, and PTO for longer-term non-core product or business model predictions.  BASES and PTO are quite complementary.

Can PTO be used for operational innovations (i.e. innovations in the way we manufacture etc., not in new products or services)?

No. The PTO currently has no capability to analyze opportunities that are not products or services sold to customers (B or C).

To the extent that operational innovations affect product or service cost (i.e. margins), performance, quality etc. then these differential effects on the product or service can be analyzed.

 In the Clorox Greenworks example, growth is predicted at 2%, what has it been?

When Clorox Greenworks was introduced, it grew to around $30M in its first few years, a growth rate much higher than 2%. It is unclear, however, how profitable it was since an extensive marketing campaign was undertaken. Since then growth has been incremental growing “around the rate of GDP”.  Current data indicates that growth is declining, meaning that 2% may have even been a bit optimistic. More information can be found in this 2013 NY Times article.

 How do we use the PTO service?

How do we choose which opportunities to run the simulation on?

The opportunities selected for simulation should be the ones that are the most critical for the company to ‘get right’. These are generally those opportunities that are the most significant for the company’s future growth and are causing the most discussion and uncertainty within the company. We do not recommend that a company run the simulation on incremental or sustaining opportunities.

Many companies choose to use the PTO service before major investment decisions need to be made. In these cases, PTO can provide another level of information for decision making purposes.

How do we collect the information to fill out the input forms?

The process of gathering the information for the simulation input involves examining documents and conducting interviews with people involved in the conceptualization and development of the opportunity of interest. Documents can include presentations, spreadsheets, written descriptions, requirements and specifications or any other description of the opportunity as it is currently envisioned.

As part of the PTO service, Inovo helps collect all relevant information about the opportunity, does the interviews and extracts what is needed for the simulation input.

What if we don’t have enough information? We don’t just like to guess at things that are so important.

Many people worry that some of the decisions asked about in the inputs (ex. what channels to use, gross margins, etc.) aren’t known yet, so they should hold off on submitting the inputs.  The beauty of the model and simulation is that it can deal with SWAGS on the inputs. This provides at least a baseline set of assumptions that can be discussed. They may be wrong, but the simulation will point that out and lead to substantive discussions about why they are wrong and what direction to go.

How do we get started?

We start with a kick-off teleconference where we discuss which opportunities should be tested and go over what information is required. We will ask for any materials on the opportunity and for introductions to the people who have been involved in the development and/or championing of the opportunity. All of the details of data entry and running of the simulation will be taken care of by Inovo.

What information do I need to provide?

Your documents and the interviews we do let us create a qualitative description of the opportunity. It could be about an early stage hypothesis, a big mature business, or anything in between. You don’t need any “big data.” Just show us the documents and do your best to tell us, in your own words, about the opportunity idea you have in mind.

What happens to the information we provide?

The inputs Inovo creates are based on the documents read and interviews conducted. They are processed by an analyst at Growth Science to harvest the data needed to run the simulation and begin the simulation process itself. If we aren’t sure about something or if it needs clarification, we’ll discuss the situation with you.

What if the inputs are wrong or if they are biased by the person providing the information?

We understand that different people in the company may view an opportunity very differently and may or may not have complete information. We try and be as complete as possible in reviewing all the documents related to the opportunity and interviewing the people responsible for the opportunity but we don’t necessarily take their word for any of the inputs. For example, you may believe your product is the lowest cost, but we may find several competitors at even lower costs. In other words, we want to know what you think you’re doing, but we don’t limit our analysis to it. We cross-check and go beyond the inputs you give us. Once in a while, however, we miss a key assumption, or the information you supply fails to mention something really important about the business. If that happens, we’ll correct the assumption and run the simulation again until it’s right.

What if I don’t have the necessary Information?

Just make your best educated guess. For example, if you don’t know the exact market size, give us your estimate. If you aren’t sure what the product’s gross margins will be, tell us what you’re aiming for. Don’t be frozen by a lack of data. Share your assumptions and we’ll help you test them.

What if the opportunity is at a really early stage and is only a vague concept?

Don’t worry about trying ideas that are very early stage, even if they’re on the back of a napkin. You just need to be willing to make educated guesses. Simulations are designed to test a set of business model assumptions, case-by-case. So if your idea is still forming or has different directions it could pursue, input whatever assumptions you have and the results can help you narrow down the best paths forward.

Can I get a copy of the filled in input form?

Absolutely. We can provide a copy of the input form at any point in time. A copy of inputs will also be attached to the final results.

 

PTO Results and Deliverables

Does it work for global opportunities (offerings sold or developed in other regions)?

The analysis works equally well for products or services sold globally or products and services developed in other countries – China, India etc. However, where markets are not competitive or are state controlled i.e. North Korea, the results will not be reliable.

One limitation is that the data harvesting tools can only harvest English characters, although the language doesn’t have to be English.  For example, Spanish, French, Chinese (translated into English characters) content can be harvested and interpreted.  Data in other scripts – ex. Chinese characters, Hindi characters, Arabic text etc., cannot be.  Almost every language has enough translated into English characters that there are usually no issues.

Can this be used to look at M&A opportunities?

Yes, PTO is used by corporates to help identify, test and validate M&A opportunities.

It works when the parent company has many business lines as well as when targets have multiple business lines, although acquirers usually have a specific product in mind when they do an acquisition (unless it is just a consolidation play).  PTO can look at each target company as a whole, but it gives more interesting results if a specific product or service within the target company is specified.

What is in the report that we get for each opportunity?

In addition to the results of the analytics simulations, the Opportunity Analysis Report (a PowerPoint slide deck) contains the following:

  • A concise summary description of the opportunity
  • An assessment of the opportunity using Inovo tools Opportunity Readiness, Should-Could and Strategic canvas
  • The results of the analytics simulation
  • One slide breakdowns of the Competitiveness, Parent Company Fit, Market Momentum and Market profitability factors affecting success
  • Relevant information discovered during simulation data harvesting
  • A summary of ‘areas for attention’ that are the focus of workshop activities.

This report provides the detailed information necessary for your company to make rational decisions on how to proceed with the opportunity and is the basis for the PTO workshop where opportunities can be shaped to be more successful.

Are the factors that contribute to an opportunity’s success or failure broken out?

Yes. As part of the PTO report you get an analysis of each of the four categories of analysis – Competitiveness, Parent Company Fit, Market Momentum and Market Profitability. The specific details for each of these is included in the PTO report.

What is the workshop? What do we do there?

The post-simulation workshop is where the insights gained from the simulation are used to shape the opportunities in ways that will make them more successful. Sometimes this is not possible and the workshop participants will end up recommending that the opportunity be abandoned. Even opportunities predicted to be successful can benefit from examination of the simulation results and discussing the factors that the simulation surface

Business Information

What PTO services does Inovo offer?

The PTO pilot project is a six week program to assess three (3) opportunities of your choice. It is designed to get you familiar with the service and to verify that it provides valuable insights. Inovo will work with you to help select the three opportunities, run the simulations, produce the reports and conduct a workshop.

The PTO service is also integrated as a component of Inovo’s opportunity discovery offering. All Level 3 opportunities are put through the PTO process before they are finalized and presented.

There are many other ways that Inovo can work with you to make use of the PTO service, all the way from occasional use for special situations to high volume use on a regular basis. Please contact Inovo to discuss your situation.

What are the advantages of working with Inovo?

Growth Science serves as the analytics engine and works with companies like Inovo to bring the benefits of the analytics to customers like you.  Inovo provides companies a full service offering that includes access to the analytics simulation along with additional research, analysis, reporting and collaborative working sessions. By working with Inovo you get the pricing, access, additional analysis and support that are not offered by Growth Science directly.