R Questions Tag Pairs on Stackoverflow

[This article was first published on Once Upon Data, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Months ago, I passed by R Questions from Stack Overflow published on Kaggle. I was interested in tag pairs in particular, i.e. which tags appear together in R questions, so I worked on this simple kernel.

This week, I had some time so I thought about deploying a simple Shiny App, to give more people access to exploring the tag pairs. So here is the App, where you can see the most frequent tags that appear with a certain tag. And below is the full code of how I processed and aggregated the data.

R Tag Pairs Shiny App



Data Aggregation

I selected the questions with more than one tag (in addition to R) and did the following:

  • Step 1: get all the tags corresponding to question ID
  • Step 2: find all pair combinations from these tags
  • Step 3: combine all pairs from all the questions in one dataframe

Step 1: get all the tags corresponding to question ID

<span class="c1"># group by question ID and nest tags
</span><span class="n">datn</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">dat</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">group_by</span><span class="p">(</span><span class="n">Id</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">filter</span><span class="p">(</span><span class="n">n</span><span class="p">()</span><span class="o">></span><span class="m">1</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">nest</span><span class="p">(</span><span class="n">.key</span><span class="o">=</span><span class="s2">"Tags"</span><span class="p">)</span><span class="w">
</span>

Now if we look at a certain question ID, we find all the tags in one list, for example for Q#79709

[[1]]
# A tibble: 4 × 1
               Tag
             <chr>
1           memory
2         function
3 global-variables
4     side-effects

Step 2: find all pair combinations from these tags

Now, we will get all the possible pairs from the questions’ tags:

<span class="c1"># map each Tag list to combn() to get all the combinations from a list
</span><span class="n">datn</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">datn</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">mutate</span><span class="p">(</span><span class="n">pairs</span><span class="o">=</span><span class="n">map</span><span class="p">(</span><span class="n">Tags</span><span class="p">,</span><span class="w"> </span><span class="o">~</span><span class="n">combn</span><span class="p">(</span><span class="n">.x</span><span class="p">[[</span><span class="s2">"Tag"</span><span class="p">]],</span><span class="w"> </span><span class="m">2</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                                 </span><span class="n">t</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                                 </span><span class="n">as.data.frame</span><span class="p">(</span><span class="n">stringsAsFactors</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">F</span><span class="p">)))</span><span class="w">
</span>

For the same question we checked in the previous step, we can see that the pairs are as follows:

[[1]]
                V1               V2
1           memory         function
2           memory global-variables
3           memory     side-effects
4         function global-variables
5         function     side-effects
6 global-variables     side-effects

Step 3: combine all pairs from all the questions in one dataframe

Now we will combine all the pairs from all questions in one dataframe and count the freq of each pair:

<span class="c1"># combine all pairs in one dataframe
</span><span class="n">dat_pairs</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">plyr</span><span class="o">::</span><span class="n">rbind.fill</span><span class="p">(</span><span class="n">datn</span><span class="o">$</span><span class="n">pairs</span><span class="p">)</span><span class="w">

</span><span class="c1"># put pairs in the same order
</span><span class="n">dat_pairs</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">dat_pairs</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">mutate</span><span class="p">(</span><span class="n">firstV</span><span class="o">=</span><span class="n">map2_chr</span><span class="p">(</span><span class="n">V</span><span class="m">1</span><span class="p">,</span><span class="n">V</span><span class="m">2</span><span class="p">,</span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">sort</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">))[</span><span class="m">1</span><span class="p">]),</span><span class="w">
               </span><span class="n">secondV</span><span class="o">=</span><span class="n">map2_chr</span><span class="p">(</span><span class="n">V</span><span class="m">1</span><span class="p">,</span><span class="n">V</span><span class="m">2</span><span class="p">,</span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">sort</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">))[</span><span class="m">2</span><span class="p">]))</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">select</span><span class="p">(</span><span class="o">-</span><span class="n">V</span><span class="m">1</span><span class="p">,</span><span class="o">-</span><span class="n">V</span><span class="m">2</span><span class="p">)</span><span class="w">

</span><span class="c1"># count the frequency of each pair
</span><span class="n">pair_freq</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">dat_pairs</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">group_by</span><span class="p">(</span><span class="n">firstV</span><span class="p">,</span><span class="n">secondV</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">summarise</span><span class="p">(</span><span class="n">pair_count</span><span class="o">=</span><span class="n">n</span><span class="p">())</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">arrange</span><span class="p">(</span><span class="n">desc</span><span class="p">(</span><span class="n">pair_count</span><span class="p">))</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
        </span><span class="n">ungroup</span><span class="p">()</span><span class="w">
</span>

Here we can see the top 40 pairs:

<span class="n">datatable</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">pair_freq</span><span class="p">,</span><span class="m">40</span><span class="p">),</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="n">pageLength</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">))</span><span class="w">
</span>


Tag-Pairs for a Certain Tag

Here we can pick one tag and see all the other tags that appear with it and the frequency of each.

<span class="c1"># Get all pairs with a certain tag
</span><span class="n">GetTagPairs</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">df</span><span class="p">,</span><span class="w"> </span><span class="n">tag</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="n">df</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                </span><span class="n">filter</span><span class="p">(</span><span class="n">firstV</span><span class="o">==</span><span class="n">tag</span><span class="o">|</span><span class="n">secondV</span><span class="o">==</span><span class="n">tag</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                </span><span class="n">arrange</span><span class="p">(</span><span class="n">desc</span><span class="p">(</span><span class="n">pair_count</span><span class="p">))</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                </span><span class="n">mutate</span><span class="p">(</span><span class="nb">T</span><span class="m">2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ifelse</span><span class="p">(</span><span class="n">secondV</span><span class="o">==</span><span class="n">tag</span><span class="p">,</span><span class="w"> </span><span class="n">firstV</span><span class="p">,</span><span class="w"> </span><span class="n">secondV</span><span class="p">))</span><span class="w"> </span><span class="o">%>%</span><span class="w"> 
                </span><span class="n">select</span><span class="p">(</span><span class="nb">T</span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="n">pair_count</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">
</span>

Example: ggplot2 Pairs

If we take ggplot2 for example, we can see that the most frequent tags that appeared with it are the following:

<span class="n">ex</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">GetTagPairs</span><span class="p">(</span><span class="n">pair_freq</span><span class="p">,</span><span class="w"> </span><span class="s2">"ggplot2"</span><span class="p">)</span><span class="w">

</span><span class="n">datatable</span><span class="p">(</span><span class="n">head</span><span class="p">(</span><span class="n">ex</span><span class="p">,</span><span class="m">40</span><span class="p">),</span><span class="w"> </span><span class="n">options</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="n">pageLength</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">))</span><span class="w">
</span>


You can see the whole list for any tag in Shiny App.

In conclusion

Pair tags give us an idea about the areas of interest, the relations between topics/packages, and the frequently used packages in the R community. We can also draw a full network to visualize more complex relations. However, these were the tags in questions posted till 19 October 2016. Definitely things change, and more tags get into the list with time. I personally expect that Tidyverse and its packages are mentioned more frequently in 2017. An updated dataset would help confirm this hypothesis!

To leave a comment for the author, please follow the link and comment on their blog: Once Upon Data.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)