Final Notes

Model 0 (Benchmark)

We need a benchmark model to see whether our own models perform better than the base model, the case in which we don’t include any explanatory variables.

$$p = \frac{e^{\beta_{0}}}{1 + e^{\beta_{0}}}$$

The expression above deals with population (real) parameters. We don’t have access to probabilities of this particular purchase event happening. We only have records of it happening or not, we’ll be using these as our dependent variable and estimate the sample estimator $\hat{\beta{i}}$ of $\beta{i}$.

These are very important concepts in statistics but we only need to make this distinction to make sure we know that the coefficients we’re getting from regression analyses are merely estimations of true values we can’t ever know. That’s why we are interested in statistical significance and confidence intervals when assessing and interpreting the model output.

Yes, the coefficients we’re getting in the regression analysis output are $\hat{\beta{i}}$.

## 
## Call:
## glm(formula = Pass ~ 1, family = binomial, data = passdf)
## 
## Deviance Residuals: 
##    Min      1Q  Median      3Q     Max  
## -1.183  -1.183   1.171   1.171   1.171  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)
## (Intercept)  0.01394    0.03560   0.392    0.695
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4375  on 3155  degrees of freedom
## Residual deviance: 4375  on 3155  degrees of freedom
## AIC: 4377
## 
## Number of Fisher Scoring iterations: 3

To see that the intercept is related closely to the average (when the effects of the explanatory variables are discounted), look at the expression below. We used the intercept coefficient in the model and the value we are getting is

$$p = \frac{e^{0.01394}}{1 + e^{0.01394}} = 0.5034$$ This value is identical to the empirical (observed) probability of the pass purchase event happening.

$$\bar{p} = \frac{1\times I[Pass = 1] + 0\times I[Pass = 0]]}{N} = \frac {1589}{3156} = 0.5034$$ The reason we’re using this model as benchmark is actually this fact: is the model we are using actually any better than simply taking the average?

Model 1

$$p = \frac{e^{\beta_{0} + \beta_{1}Promo}}{1 + e^{\beta_{0} + \beta_{1}Promo}}$$

The model that we’re estimating is shown above.The Promo variable is the dummy variable. It equals 1 when Promo is available, 0 otherwise.

## 
## Call:
## glm(formula = Pass ~ Promo, family = binomial, data = passdf)
## 
## Deviance Residuals: 
##    Min      1Q  Median      3Q     Max  
## -1.262  -1.097   1.095   1.095   1.260  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -0.19222    0.05219  -3.683 0.000231 ***
## PromoBundle  0.38879    0.07167   5.425 5.81e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4375.0  on 3155  degrees of freedom
## Residual deviance: 4345.4  on 3154  degrees of freedom
## AIC: 4349.4
## 
## Number of Fisher Scoring iterations: 3

Remember that this is what we need to interpret the model, as this is the most simple linear relation we can get. In order to interpret probabilities, we’ll need another approach.

$$odds = {e^{\beta_{0} + \beta_{1}Promo}}$$ So, basically when Promo = 1, this equation becomes the expression below. We can see that the intercept has changed. We can also see that $e^{0.388}$ is a factor that multiplies the base odds ratio (in the context of this particular model!). We know that $e^{0.388} = 1.47403 $ and that it means the being exposed to promo is associated with 47% increased odds of purchasing the season pass.

$$odds = {e^{-0.192 + 0.388}} = e^{0.388}e^{-0.192}$$

Model 2

Now pay attention to what’s happening below. We are dealing with our categorical variable with 3 levels (Mail,Park,eMail) and that we cannot observe eMail anywhere in the expression below! No worries, nothing is wrong here. But we need a different perspective here to interpret the results we have received.

$$p = \frac{e^{\beta_{0} + \beta_{1}Promo + \beta_{2}Mail + \beta_{3}Park}}{1 + e^{\beta_{0} + \beta_{2}Mail + \beta_{3}Park}}$$

One thing to note here is that Mail and Park cannot be equal to 1 at the same time because they are mutually exclusive by definition! Since the Channel variable could take up only 1 of mail, park and e-mail at a time, the same logic applies here as well. So, when both mail and park variables equal to 0, that means the active channel is the e-mail channel!

But the most important thing is the following, let us calculate the odds for the case that we have promotion bundle and we are using the mail channel.

$$p = \frac{e^{-2.078 -0.5602\times 1 + 2.1761 \times 1 + 3.72 \times 0}}{1 + e^{-2.078 -0.5602\times 1 + 2.1761 \times 1 + 3.72 \times 0}}$$

## 
## Call:
## glm(formula = Pass ~ Promo + Channel, family = binomial, data = passdf)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.9079  -0.9883   0.5946   0.7637   2.3272  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -2.07860    0.13167 -15.787  < 2e-16 ***
## PromoBundle -0.56022    0.09031  -6.203 5.54e-10 ***
## ChannelMail  2.17617    0.14651  14.854  < 2e-16 ***
## ChannelPark  3.72176    0.15964  23.313  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4375.0  on 3155  degrees of freedom
## Residual deviance: 3490.2  on 3152  degrees of freedom
## AIC: 3498.2
## 
## Number of Fisher Scoring iterations: 4

$$odds = {e^{-0.2078 -0.5602 +2.1761 }} = e^{2.161}e^{-0.5602}e^{-0.2078}$$ So, compared to the case where no promo using the channel e-mail, using promo and the mail channel is associated with 5.03 times increased odds of seeing a pass purchase decision. Because $ e^{2.161}e{-0.5602} = 5.0324$.

We should always be careful what we are comparing our interpretation against, as the base case is only implied when we are using dummy variables.

In order to calculate the probability corresponding to the particular case above, we’ll be using the p formula. This gives us a probability of 0.386, which seems to be worse than doing nothing (compare this against model 0, where the predicted probability was 0.50)

Model 3

Model 3 is only different in that it includes interaction terms. We use interaction terms to see if there is an amplified or a dampened effect when we are using more than two treatments at once. There may be an unintended interplay between, let’s say, using Promo through the park channel.

$$p = \frac{e^{\beta_{0} + \beta_{1}Promo + \beta_{2}Mail + \beta_{3}Park + \beta_{4}Promo \times Mail + \beta_{5} Promo\times Park}}{1 + e^{\beta_{0} + \beta_{1}Promo + \beta_{2}Mail + \beta_{3}Park + \beta_{4}Promo \times Mail + \beta_{5} Promo\times Park}}$$

We can see that the coefficients related to interaction effects are significant. This tells us that some other mechanism is at play here. Obviously, the interaction effects will not be available if Promo is not active as the dummy variable corresponding to Promo will be equal to 0 (The same applies to the E-mail channel).

## 
## Call:
## glm(formula = Pass ~ Promo + Channel + Promo:Channel, family = binomial, 
##     data = passdf)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.9577  -0.9286   0.5642   0.7738   2.4259  
## 
## Coefficients:
##                         Estimate Std. Error z value Pr(>|z|)    
## (Intercept)              -2.8883     0.1977 -14.608  < 2e-16 ***
## PromoBundle               2.1071     0.2783   7.571 3.71e-14 ***
## ChannelMail               3.1440     0.2133  14.743  < 2e-16 ***
## ChannelPark               4.6455     0.2510  18.504  < 2e-16 ***
## PromoBundle:ChannelMail  -2.9808     0.3003  -9.925  < 2e-16 ***
## PromoBundle:ChannelPark  -2.8115     0.3278  -8.577  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4375.0  on 3155  degrees of freedom
## Residual deviance: 3393.5  on 3150  degrees of freedom
## AIC: 3405.5
## 
## Number of Fisher Scoring iterations: 5

$$p = \frac{e^{-2.88 + 2.10Promo + 3.1440Mail + 4.64Park -2.98Promo \times Mail -2.81 Promo\times Park}}{1 + e^{-2.88 + 2.10Promo + 3.1440Mail + 4.64Park -2.98Promo \times Mail -2.81 Promo\times Park}}$$ Let us compute the probability of using the e-mail channel as part of a bundle. Due to both Park and Mail being equal to 0, these are terms we are left with

$$p = \frac{e^{-2.88 + 2.10Promo}}{1 + e^{-2.88 + 2.10Promo }} = 0.315$$

We can see that using e-mail as part of a bundle is actually not a good idea.

On final note about confidence intervals:

We can see below that the confidence intervals for the odds are quite wide. This is not surprising as the model is not trying to estimate the odds ratios but rather the coefficients in the generalized linear model (in our case, logistic regression). You can see that, none of the significant variable include 1 in their confidence interval (which is the ineffective element when thinking about odds ratios.)

##                               2.5 %       97.5 %
## (Intercept)              0.03688720   0.08032263
## PromoBundle              4.78970184  14.31465957
## ChannelMail             15.54800270  35.97860059
## ChannelPark             64.74364028 173.57861021
## PromoBundle:ChannelMail  0.02795867   0.09102369
## PromoBundle:ChannelPark  0.03135437   0.11360965