Our reviews process consists of different steps. We want to explain them one by one in this page. Please note that we often refer to our reviews with the term “analysis”, as we apply sensory analysis guidelines to our work.
Our process was created together with our sensory science consultant when we started Cigar Sense and, being an agile company, we strive to make it better on an ongoing basis.
The objectives of our cigar reviews
At Cigar Sense we establish cigar profiles to meet our core objective of guiding cigar lovers to identify which cigars best fit their unique personal preferences. These can be either sensory (aromas, tastes, tactile sensations, nicotine strength and other secondary preferences) or more straight forward, such as price, wrapper type, ring-gauge,… The objective information we provide with our cigar analyses is predictive: in average, our active members, are delighted by, or like, 9 out of 10 cigars we recommend to them.
Our cigar analyses aim to also educate and inform the cigar lover through independent, objective data. This means that when someone else will buy and taste another sample of the same cigar we describe, s/he will very likely find the descriptors we talk about when smoking the cigar.
Our cigar profiles are established by consolidating the analyses of trained panelists and feed one of the two main databases based on which Cigar Sense works: the one containing detailed cigar profiles, which are matched with our members’ detailed personal preferences.
Scores and ratings
We match the data from these two data sets to obtain one score, the fit %, which enables us to rank, for each of our members, which cigars are the best match for them. The fit %, in fact, represents how much a cigar fits the personal preferences of a given member.
All we do is based on the fact that there is no good or bad taste, there is no best or worst cigar. Only our personal taste counts. Of course, there are low quality cigars, we discuss this in the last paragraph of this page.
We do not publish any generic score for cigars, as generic scores are not predictors of whether a consumer will like the cigar or not. Our method has an accuracy rate of 90%+. This means that our active members, in average, are delighted by, or like, 9 out of 10 cigars we recommend to them.
The procurement of cigar samples
We don’t always obtain samples from manufacturers for our analyses. When we don’t, we procure samples at different retailers.
We currently test 10 samples of each cigar.
We test blind. This means that we remove the labels from the cigars we need to test and re-band them with a numbered label.
The tests are performed by our team of panelists, who are continuously trained to improve their sensory acuity and, especially, to become increasingly aware of the many biases that can affect a review. The panel performance is reviewed an monitored.
Our objective is to achieve repeatability for most panelists. This means that, if we add another sample of the same cigar, and this last has an acceptable production consistency, there will be agreement in further test outcomes performed under the same test conditions, even if the panelist is not the same.
Our panel is composed of 6-8 panelists who have reached a higher level of reliability and are measured and monitored for repeatability. We may perform replica with these panelists. An additional number of panelists are either under initial training or are less involved in our training program. The samples distribution takes all these factors into account. Obviously, the more we rely on the same trained panelists, the more consistently the panel performs. If a single panelist consistently disagrees with most attributes measurements by the rest of the panel, this panelist should be re-trained or is not fit for our panel.
There are additional reasons for which a panelist is considered as not fit for our panel. These are mainly related to the commitment to our process and method.
In total, today, there are 13 panelists in our team.
The recruitment of panelists
We have a multiple-step onboarding process for our panelists:
- familiarize with the Cigar Sense service to our members, to understand the role of our analysis process
- a seminar covering
- cigar analysis process
- sensory analysis principles
- guidelines, including the priming of our buds and of our mind, key errors to avoid,…
- a questionnaire
- ‘on the job’ training
Our panelists have different backgrounds and represent different world regions.
The ongoing training of our panelists
The training occurs in two different ways:
- the use of aroma standards (also called aroma tests) to calibrate thresholds and train the identification and memorization of aromas
- monthly meetings where various topics are discussed
- ad hoc one-one interactions as needed
Opinions on cigars are not discussed during our meetings. We obviously all have our personal preferences, like anybody else. We rather discuss about how the panel is aligned and the awareness of biases found through the data.
The samples distribution
Samples are shipped or handed over to our panelists based on each panelist capacity. Capacity is the number of samples each panelist can test each month for Cigar Sense.
Due dates are set for every batch and for every aroma test.
Humidification is provided in each batch and due dates take into account the carrier turnaround time. In addition, each panelist decides how long it is required to let the samples further settle in their humidor.
The analysis form is available online. It consists of 5 pages which guide the panelist to qualify and quantify the various descriptors that will contribute to the cigar profile.
The reveal of the blind test is available immediately after submission of the form, on the panelist dashboard.
The panel performance monitoring
We observe the degree of alignment between the descriptors measurements by each individual panelist against the ones of the panel.
For replica, we measure repeatability as the degree of homogeneity between replicated tests.
In addition, it’s not rare to discover biases by looking at the panelists data. For example, in the past, we have been able to uncover and correct a key bias and identified the panelists who needed re-training on how to differentiate pepper aroma from spiciness. This difference might be irrelevant to a leisure smoker, for a recommender system it is of absolute importance to address it and fix it.
The consolidation of the individual cigar analyses
The consolidation of each individual analysis data occurs blind. This means that the person who consolidates the data does not know what cigar the data relates to. This is important for us to reduce the leniency bias which otherwise highly influences so many published reviews in the cigar world: we all want to be kind to our clients and friends, this is human.
We have an internal quality threshold. Any cigar that, after data consolidation, does not pass such threshold, is not recommended to our members. The relevant data will be visible to our internal members with a caveat informing that additional samples are be required to define a final profile. We always give an additional chance to a cigar.
The publication of cigar profiles
Upon the upload of the consolidated analyses, the new cigar profiles are available to our members:
- in their recommendations, if their personal preferences match at least 80% of the cigar characteristics
- in the search database
We also publish cigar analyses to a wider audience who is not required to have a membership. The number of these published analyses is far lower than the analyses available to our members.
Updates of cigar analyses
We may update a cigar analysis for different reasons:
- a cigar is not recommended and we add samples to verify if it can be
- a cigar is the object of a “false positive”, meaning we recommend the cigar to one of our members, but the cigar is not liked by such member. 1 out of 10 recommended cigars may in fact not be liked by our members. This can happen for multiple reasons:
- the cigar was a dud
- the cigar construction was bad
- the flavor and strength does not correspond to the member’s profile – in this case we take the opportunity to guide the member to review the personal profile and, if no explanation for the dislike is found in the member’s profile, we consider the dislike a false positive.
The communication with manufacturers
We may be in touch with manufacturers for different reasons. When it comes to cigar analyses, this is our process:
The manufacturer provided the samples
If the manufacturer provided samples to us, we always inform them of the readiness of the analysis.
The outcome is positive
If the outcome is positive, meaning the cigar is recommendable to our members, we communicate the results and discuss if there are questions.
If we publish the cigar analysis to a wider public, we tag the manufacturer on social media.
The outcome is negative
If the outcome is negative, meaning the cigar is not recommendable to our members, we inform the manufacturer, explain the reasons for the negative outcome and make ourselves available to discuss further. We are always willing to add samples and give the cigar another chance. As we monitor our repeatability goal, we also observe, however, that adding samples does not dramatically change the picture.
Currently approximately 15% of the analyzed cigars do not pass our quality threshold and cannot be recommended to our members or be the object of public analyses unless we add samples and retest them.
We purchased the samples
When the manufacturer did not provide the samples, we always like to communicate with them about the results of the analyses. However, given the very high flow of reviews that the manufacturers are exposed to today, we normally only tag them in social posts if we publish the analysis to a wider public.
Occasionally we have informed manufacturers of cigars that are not recommendable to our members. The result has often been the manufacturer sends us additional samples.
Given the fact that we never publish analyses to the wider public for cigars that are not recommendable, the risk for brands to be negatively exposed in public by us is practically inexistent.
We are thankful to our members and to those manufacturers who trust our method.