In the recent Directors’ Roundtable on Film Companion (video below), filmmaker Karan Johar spoke extensively about using Ormax Media’s research tools for films produced by Dharma Productions. As a result, we have received several queries related to our Content Testing work, from various constituents of the Indian entertainment business. This explainer attempts to answer 14 frequently-asked questions in this regard. We understand that you may have other queries too. We will be happy to connect with you and answer them.

1. What is the difference between focus screenings, test screenings, focus groups and other such terms one keeps reading about?
These terms are often used interchangeably. Without getting into technical distinctions, it can be said that all such terms refer to market research being done to test content, such as a movie or a series. We like to refer to this type of work collectively as Content Testing.

2. Why does one even need Content Testing?
Testing your content can be useful in various ways. In all our Content Testing work, we report the Ormax Power Rating (OPR) of the tested content (film, web-series, TV show, animation, etc.). This score, on a 0-100 scale, encapsulates the audience approval or rejection of the content, e.g., a score of 70+ suggests the audience have whole-heartedly accepted the content, while a score below 40 suggests outright rejection. Along with OPR, the report provides detailed feedback on various aspects related to the content tested, such as characters, plot points, pace, comprehension, relevance, uniqueness and many other such areas. Our proprietary analytics models are deployed on this data to forecast the following business metrics:

a. The Lifetime Box Office (LBO) potential of a theatrical film
b. Lifetime viewership potential of streaming content
c. Steady-state TVR of a TV show

As a result, a Content Testing report can help a producer, distributor or marketer take decisions of the following nature:
a. Whether to release a film theatrically or direct-to-OTT?
b. At what price point should a film be sold to a distributor or an OTT platform?
c. Whether to acquire a ready film or not?
d. Whether to green-light a concept (or script) or not?
e. At what scale to distribute and market the content?
f. How, and to which core TG, to position the content?
g. In which time slot should the TV show be aired?

Producers can use feedback from the test to make changes to the content, to improve its OPR, and hence, its business potential. We will be happy to share a few case studies with you when we connect.

3. How many films, shows and scripts have you tested so far?
We have tested 128 films and 88 film scripts till date, across Hindi, English, Tamil, Telugu, Kannada, Malayalam, Punjabi, Marathi, Bengali and Gujarati languages. We have tested 45 OTT shows and 215 TV shows till date, at concept, script or pilot episode stages, across the languages mentioned above.

4. How do you test a story or screenplay, i.e., when no video content is available?
We test scripts (at a step outline or screenplay stage) in a 'radio theatre' format, whereby the script is narrated using a voice-over. A narration of this nature retains key plot points and dialogues, and is typically about 60-75 minutes in duration for a film or a web-series, and about 15-30 minutes in duration for a GEC show. We do not test animation content or non-fiction content in this format, as video stimulus is essential for those tests. In case of remakes, we sometimes prefer testing the original film with subtitles as a substitute to the radio theatre format. The degree of changes in the adaptation vis-à-vis the original is a factor of consideration in this decision.

We have often been asked if this method of script testing can give accurate results. In our experience, we have seen very high resonance between the results of a script test and the performance of the actual content (see #9 below). Audience react to characters, emotions and key plot points, and have the ability to visualise the film or the show in their minds when they listen to an audio narration. Our audio narrations are generally written and recorded by us, to ensure a standardised format of writing and story-telling, which we have established based over more than 150 projects of this nature.

5. How do you test a series or a TV show, where the length of content is several hours?
For web-series, we typically test the first three episodes, which is a fair indicator of the show’s content strength.

For GEC shows, a concept narration of 15-30 minutes is typically used. Even when the first episode (pilot) is available in video format, we follow the screening of the episode with this concept narration, to capture the longer arc of the show, ideally covering 4-6 months of story.

6. Where is the research conducted?
For theatrical films, Content Testing research is conducted in a preview theatre. In cities where such a facility is not available, a private venue, such as a banquet hall in a hotel, is adapted to replicate the theatrical experience, with a big screen and good quality audio.

For web-series, TV shows and film scripts, commercial venues like banquet halls are used, with quality seating, video and audio devices. Large-screen flat TVs are used to play videos in such cases.

7. How do you select the test audience?
Over the last 14 years, Ormax Media has built a repository of industry data that captures the gender, age and market profile of core audience of various categories in various languages. Based on this data, we recommend a research design to the producer or platform commissioning the project, and lock the final design in consultation with them. Depending on the scale of the project and the budgets, a project may cover a sample size of 250-1,200 audience, across 2-6 markets.

Ormax Media runs a large online panel of users, who are invited to Content Testing projects if they meet the selection criteria. In every project, about 20-30% audience are recruited from outside the panel (via field research methods), to ensure that the panel keeps growing in size. No panelist can attend more than one research in one category (theatrical/ OTT/ TV) in a period of six months. The panel is incentivised (e.g., Amazon and Bookmyshow gift vouchers), in line with standard practices with such panels globally.

8. How much time does it take to complete the research and get the results?
Depending upon the sample size and the number of markets, it can take 2-4 weeks from commissioning the project to receiving the report.

9. How do you assess the accuracy of Content Testing, and what has the accuracy been so far?
For films and shows that release, we compare the actual performance to the forecasted performance as a measure of accuracy. For example, if a film was forecasted to do a business of Rs. 125 Cr but actually did only Rs. 105 Cr, its deviation is 20/105=19%, and hence, its Accuracy is 81%. This calculation, when done at a cumulative level for all films (or shows) tested, gives us the overall Accuracy of our testing products, which is typically in the 80-90% range. The chart below summarises the box office accuracy for films tested by us till date.

10. What is the cost of conducting a Content Testing research?
For the value it offers, cost of Content Testing research is very low, typically being 0.25-1.00% of the cost of production of content. The cost depends on the sample size, the choice of markets & TG, the type of research (video vs. script, film vs. OTT vs. TV, etc.), and other such factors. A project can cost between ₹ 6-25 lac ($8,000-34,000), depending on these factors.

11. Do you need the final product, with the final VFX and background music, for testing?
No. We routinely test video content in parallel with the post production process. A rough edit, with reference background music and work-in-progress VFX, is testable in most cases. As long as the material is comprehensible by the audience, test results tend to reflect an accurate picture, for the reasons mentioned in #4 above. We will recommend a test only if the content passes our internal assessment of being ready for research.

12. Test screenings have been done informally by many producers over the years. How is the Ormax method different?
Informal test screenings, including friends & family screenings, are often plagued by sample selection bias. Apart from being a statistically-rigorous process, our research tools use our proprietary forecasting algorithms that enable sound business decisions (see #2 above), which make them a lot more scientific and rigorous than informal methods. Our analysts work on Content Testing projects for a living, and bring their cumulative experience across several projects on board every new  project.

13. Does one need to worry about their content getting leaked during Content Testing research?
Across almost 500 Content Testing projects conducted by us till date, there has not been a single instance of any leak. We follow a strict ‘Privacy & Security’ protocol, which is summarized in the chart below. For films and high-profile shows, we never take physical possession of the material (e.g., the hard disk) ourselves. We will be happy to answer any questions you may have on the protocol.

14. Can a maker attend the research to get real-time feedback on their content?
We welcome producers, directors and writers to attend the research, experience the content with the audience, and watch the focus group discussions using a CCTV setup. However, we need to be mindful of following certain protocols in this regard, especially if the people attending are celebrities, i.e., recognizable by the audience. In any case, we provide video recordings of all focus group discussions to the commissioning producer.