A history of question migration in tables and a graph

Question

[Note: Most of this post was written before the name change and I haven't changed to Software Engineering. I think that's appropriate, since I'm looking backward to when this site was Programmers.]

[Note: This in response to a comment by gnat on Meta Stack Exchange asking for information regarding migrations to and from this site. It's being shared here since it may be of interest to the community.]

I wrote a proposed philosophy of question migration. My thesis is that migration ought to be thought of as a convenience for the OP when they ask on the wrong site rather than a tool to manage content. Given there exists a chatbot to detect mentions of programmers.SE in order to deflect potential migration/crossposting, it seems to me that this site is particularly concerned about migration.

As a baseline, here are some statistics for the entire history of the site going back to the very first question asked:

Total Questions 
--------------- 
98768           

Migrated Direction 
-------- --------- 
 8900    out       
14484    in       

Exported to:                      count  good 
------------                      -----  ---- 
stackoverflow.com                  6482  6030 
workplace.stackexchange.com         297   273  
codereview.stackexchange.com        264   228  
superuser.com                       216   203  
security.stackexchange.com          173   171  
ux.stackexchange.com                165   162  
meta.programmers.stackexchange.com  125   111  
dba.stackexchange.com               118   115  
pm.stackexchange.com                103    97   
cs.stackexchange.com                 87    81   

Imported from:                    count  good  
--------------                    ----- ----- 
stackoverflow.com                 13828 11630 
codereview.stackexchange.com        300   272   
superuser.com                        45    34    
serverfault.com                      40    32    
pm.stackexchange.com                 31    28    
cs.stackexchange.com                 31    19    
webmasters.stackexchange.com         27    16    
workplace.stackexchange.com          26    22    
english.stackexchange.com            14    12    
ux.stackexchange.com                 14    12    

I'm just including at the top ten sites as there's a noticeable drop off in quantity after the top slot. As you can see, Programmers has received 5584 more questions from other sites (mostly Stack Overflow) than it has migrated away. A full 14% of the questions that have ever existed on this site began life on Stack Overflow.

But those all-time numbers hide an important story. Almost all of the migrations occurred before 2012:

In recent years, Programmers has been a net exporter of questions:

Total Questions since         
--------------- ------------- 
76426           2012-01-01 00:14:59


Migrated Direction 
-------- ---------
6629     out       
2307     in        

Exported to:                       count good 
------------                       ----- ---- 
stackoverflow.com                   3547 3116 
codereview.stackexchange.com         151  118  
workplace.stackexchange.com          128  104
superuser.com                         68   59   
meta.programmers.stackexchange.com    63   51   
cs.stackexchange.com                  56   50   
ux.stackexchange.com                  53   52   
dba.stackexchange.com                 48   45   
security.stackexchange.com            41   39   
unix.stackexchange.com                32   28   

Imported from:                     count good 
--------------                     ----- ---- 
stackoverflow.com                    870  584  
codereview.stackexchange.com         179  158  
serverfault.com                       23   16   
cs.stackexchange.com                  22   10   
webmasters.stackexchange.com           8    5    
workplace.stackexchange.com            8    4    
pm.stackexchange.com                   7    6    
ux.stackexchange.com                   6    6    
superuser.com                          6    4    
webapps.stackexchange.com              4    3  

I picked January 1, 2012 because it's a handy cut off, not because of any particular event on the site. The quality of incoming questions has been respectable:

Migrated here Avg score Avg answers Avg comments Closure % Deletion % Avg age (days)
------------- --------- ----------- ------------ --------- ---------- --------------
         2307       6.0         2.6          3.5      14.3       16.0           32.3   

Do remember votes from the source site are reflected in the score. For instance, questions migrated from Stack Overflow to this site had an average score of 5.8 and 1.1 answers. So score might be less useful a measure of quality than the fact that these questions got new answers after migration and are relatively less likely to get closed than organic questions.

That said, fully a third of the questions migrated from Stack Overflow have been closed or deleted. We don't have an easy way to quantify crossposted questions or questions asked at the prompting of misinformed commenters on another site.

Answer 1

I don't think that there's anything surprising in this data.

---

The Importing of Questions from Stack Overflow

The high number of imported questions from Stack Overflow makes sense. Software Engineering was originally Programmers, which was originally proposed as Not Programming Related. The proposal for Not Programming Related was for a safe place to ask the questions that were too centered on opinions or too broad for Stack Overflow. That didn't last very long, and the quality of questions led to a blog post on the Stack Overflow blog. This site was one of the main reasons for the Good Subjective, Bad Subjective and Real Questions Have Answers posts, and numerous Meta discussions about question quality.

This transition from Not Programming Related to something pretty close to what we are today spanned from the proposal on 3 June 2010 through the private beta that started on 1 September 2010 through a lot of the public beta which began on 8 September 2010. However, by our full launch in early December 2010, we had changed our scope to something closer to what it is today. This turmoil in definition was about 6 months, but wasn't clear to anyone who didn't participate on the Meta site.

One big problem with all of this turmoil was that, until a few days ago, we continued to have the same name and branding. To anyone outside of active participants on our community, they may not have been aware of the drastic change in expected question quality or scope. This is why Duga exists - to help us find people (primarily on Stack Overflow) who are blindly suggesting our site without understanding our scope. The things that should have made our scope change blindingly obvious to anyone - new subdomain, new logo - were flat-out rejected in June 2012.

Until June 2012, Stack Overflow users could vote to migrate questions here. When that Meta post was written, we had a 26% rejection rate for migration. The history of migration from Stack Overflow to Programmers was so bad that, in December 2010, we had to ask Stack Overflow users to stop using Programmers as their toilet bowl. Even with us seeing the influx of garbage sent from Stack Overflow, it still took a year and a half to stop individual users from migrating questions.

Also in October 2012, migrations for questions older than 60 days were disabled.

2012 was a big year for stopping incoming migrations. It stopped users from sending old (and likely low quality) SO questions to us by taking away their ability to migrate to us and even stopped SO mods (who may not have understood our site) from migrating old questions to us.

---

The Exporting of Questions to Other Sites

The high number of exported questions also makes sense. A general trend in increase of migrations also makes sense.

Since 2010, our scope has been stable, for the most part. When we went live in December 2010 and were introduced to the world in a blog post, the things that were on topic here included software engineering, developer testing, algorithms and data structures, design patterns, architecture, development methodologies, quality assurance, software law, and business concerns.

Really, what has always been on-scope here is software engineering. Most of the things in the original list of things on-topic are part of software engineering. Today, we have a more succinct list of things that are on-topic, but it's still essentially all of software engineering.

However, some aspects of software engineering are better covered on other sites. Information Security, Database Administrators, Writers, Project Management, Computer Science, The Workplace, Law, and Open Source were all started after Programmers was, and these are all things that someone working in software engineering would learn, either as part of a curriculum or on-the-job. It's not that database design questions or data structure and algorithm questions or licensing questions are off-topic here, it's just that there are communities who can give better answers to these types of questions.

I think that part of this exporting culture may have been on us, as a community. When we shifted from the original Not Programming Related concept to Programmers, I think that we kind of normalized around the idea that we were actually a community of software engineers. Being primarily professionals, we wanted a site where the questions we had as professionals would be accepted without being lost in a sea of "fix my code plz" questions on Stack Overflow. There were already sites for writing and debugging code, professional-level system administration, using various computer software tools, mathematics, statistics, user experience, and a few other things that we cared about. So we include a variety of our professional concerns in the list of things that were on-topic. Until The Workplace came around, we had more tolerance of education and career questions. Before Computer Science, we took in some of the more theoretical/mathematical algorithmic questions. Before Law and Open Source, we took in a lot of law and licensing questions (but began to constrain that since we realized we weren't lawyers and couldn't answer all of these questions).

There are two good examples for migrations. First, we did a little experiment where the same question was asked on multiple sites. The answers were dramatically different. Here, they were very practical, focused on the quality attributes of the software (testability, readability, performance), and written in more plain English. On CS, they were more mathematical and focused on the theoretical behavior of the algorithm. Another example is Open Source and Law - the people there have learned more about and worked with copyright and licensing and the quality in answer is so much higher - just look at the answers here vs there in software license questions.

Ideally, we should be seeing this trend downwards. It may rise slightly if a new related site goes into a public beta or launches, but shouldn't really spike. At this point, we have a well-defined scope that focuses on the core aspects of software engineering without the tangential things. The main reason for this is that a lot of these tangential things now have their own communities in various states of development.

---

The Purpose of Migration

I disagree with the thesis that "migration ought to be thought of as a convenience for the OP when they ask on the wrong site rather than a tool to manage content".

Migration is a tool to manage content, just like deletion. However, unlike closure or deletion, it preserves the content and signals to the user "yes, you asked a good question, but although we can't give you a great answer here, you can get a great answer over there". By moving the good question, we help the asker get a solution to their problem and introduce them to a community of experts if they have a similar problem in the future. We help people who have the same question in the future have access to higher quality answers. We also add one more good question to another site that will help boost its discoverability by searching.

As long as we continue to uphold the rule of "don't migrate crap", we will probably be an exporter of questions. Our new name gives people a good impression of what we are about, but they may not be aware of niche communities that would be better for very specific questions. They may also not be aware of nuances between some communities, such as Software Engineering and Computer Science and the differences in answers.

Personally, I wouldn't mind seeing migration go away. A question is either on-topic or not and it doesn't need to be our concern if they get the best answers or not. We can use comments and other tools to let people know about other communities that exist. However, as it stands now, I have a tool that helps me ensure that the user gets the best possible answers for their questions, so I use it.