[This article was first published on numbr crunch - Blog
, and kindly contributed to R-bloggers
]. (You can report issue about the content on this page here
Want to share your content on R-bloggers? click here
if you have a blog, or here
if you don't.
Logistic Regression Continued
I’m finally getting back to tackling the Titanic competition. In my last entry, I had started with some basic models (only females live, only 1st and 2nd class females live, etc), and then moved onto logistic regression. My logistic regression model at the time was not performing that well but I was also only using four features. The forums on Kaggle
contain a lot of great information. For example, I found that lots of other people were using logistic regression, and that feature engineering was what helped them increase their model performance. I’m going to describe what I did below, you can also follow along on Github
So Let’s begin with the feature engineering that I did:
- Replaced NAs in test$Fare with the median test$Fare of the remainder values
- Created a new feature called “Family” which tallies the total individuals per family by SibSp + Parch + 1
- Converted Fares to ranges (only used in logistic regression) from 1 through 6 by $20 increments, where range 6 is any Fare greater than $100
- Used the Name feature and extracted people’s titles, put them into a new feature called Title
- Replaced NAs in the Age feature with the average age of individuals based on their Title
Once all the feature engineering was complete, I used the updated features as part of a new basic logistic regression model to see my performance.
This entry gave me a score of 0.78947, which is a significant improvement from the original logistic regression model without any feature engineering. At this point, my options to improve performance from logistic regression would be to tweak the model parameters or do some additional feature engineering. Since I haven’t had much success with parameter tweaking and I’m tired of feature engineering, I’m going to move onto another model.
After spending so much time on the message boards, I found that others were using decision trees as another classification option. Two common packages used for decision trees in R are “rpart” and “party”. The “rpart” package is pretty simple to use with the data as it exists.
This achieved a slight improvement in my score as compared to my logistic regression model. My goal at this point was to learn how to boost or bag the decision tree model to see what kind of performance increase I could get. My explanation for boosting and bagging won’t be as good as this one
. I tried to understand the documentation for the “adabag” package (boosting and bagging algorithm package) in R but I could not get it to work. I went back to Trevor Stephens’ webpage and found that he used a package called “party” to run an ensemble decision tree model. This model does require some tweaking to the data prior to running. The number of factors for any features that are used must match for both the test and train sets. See my code below.
Hooray! I eeked out a slightly higher score!! Well, my original goal was to stop once I reached 0.80 but I’m fine with rounding 0.79904 up, which puts me in the top 25% of submissions. At this point, the time I am spending improving my models and updating my features is producing diminishing returns, and there are plenty of other data problems I would rather tackle. I really enjoyed playing around with the data and the models and seeing what worked and what didn’t and learned a lot doing it. One of the scariest things I learned was how little I actually understood about the models I was running. I have a basic understand of linear regression, logistic regression, and decision trees, but I don’t always know how to improve my performance if I get stuck. I think a lot of this comes from understanding the models really well so I can tweak parameters or decide to use an alternate model based on the data. Anyway, I think this can be a topic for a future post. Until then…