37

The site was successfully renamed to SoftwareEngineering. Great. The primary goal of the name change was to make it clear what this site is about, and what is off-topic here.

I'm very positive about the idea itself of changing the old name, as well as the new name selected by the community. I find that, indeed, it makes it more obvious that some questions are off-topic.

However, I noticed no decrease whatsoever in the number of questions which are obviously off-topic. But this is only my personal impression, which may be wrong.

What do statistics say? Was there a clear drop in questions closed as off-topic since the name change? Is it too early to ask?

5
  • 6
    That's interesting. My "gut feeling" observation (not backed my numbers or hard data) is that off-topic questions have slightly decreased. But, like you, I wonder what the hard data says. I wonder if the name change has changed the kind of offtopic questions we're getting too.
    – MetaFight
    Dec 5, 2016 at 13:14
  • 2
    It may be too early yet to get good statistics off of SEDE, however I may have time to jump on later today and write a query that might give some insight.
    – maple_shaft Mod
    Dec 5, 2016 at 16:33
  • 2
    "programmers" used to be the water cooler site. Then it was forced to be serious. Now the transformation is complete.
    – Thorbjørn Ravn Andersen
    Dec 6, 2016 at 12:26
  • But the new design is very pretty.
    – Thorbjørn Ravn Andersen
    Dec 6, 2016 at 12:34
  • I expect Hawthorne effect to be still going, maybe you should wait a year and compare by periods (month by month, probably) the general quality.
    – Braiam
    Dec 12, 2016 at 13:59

4 Answers 4

Reset to default
24

The close questions statistics don't allow you to see windows that you can adjust, but I wonder if you can use SEDE to explore more. I'll also see if a CM can run some queries and do some data analytics.

Here are the statistics for percentage of total questions closed, by reason, in the last 30 days:

  • Assistance in explaining, writing, or debugging code: 18.71%
  • Find or recommend stuff: 19.01%
  • Legal advice or aid: 1.17%
  • Career or education advice: 9.36%
  • Other off-topic: 4.09%
  • Customer support: 6 questions

I took a look at a 90 day window, which starts before our name change, but includes the period after the name change:

  • Assistance in explaining, writing, or debugging code: 29.55% (split across two reasons due to a wording change)
  • Find or recommend stuff: 15.58%
  • Legal advice or aid: 0.44%
  • Career or education advice: 7.61%
  • Other off-topic: 4.61%

We're trending downward in the explaining, writing, and debugging code questions. This is a very good thing.

There's a slight up-tick in finding and recommending, career/education advice, and legal advice/aid. I'm willing to discount the uptick in legal questions. We've been slightly more discerning in what goes here vs what goes to Open Source or Law. However, the fact the find/recommend and career/education is something that we should take a look at.

Shog was able to share one chart and gave permission to post it here. He didn't have any more time to do more queries or analysis, but notice the downward trend in October/November 2016?

Software Engineering Question Closed Stats

I'll ping a CM, though, and see if they can't do any cooler data analysis with pretty charts and such. I'd be interested to know what we've fixed and what needs work.

14
  • 1
    Thanks for the update.
    – MetaFight
    Dec 5, 2016 at 15:28
  • 12
    The uptick of non-debugging close reasons is because the quoted percentages are fractions of total closed questions, not the fractions of total asked questions – the UX of the 10k page is quite confusing here. The absolute numbers of all have gone wayyy down. Because the debugging questions were reduced most dramatically, the other slices of the closed-questions pie are larger in proportion. These percentages just tell us that we currently close as many questions for debugging help as we close for recommendations, which is great considering how few those are.
    – amon
    Dec 5, 2016 at 17:28
  • Don't the absolute numbers (number of questions closed for reason X) matter more than the percentages?
    – D.W.
    Dec 7, 2016 at 6:20
  • @D.W. Probably, but there's no way for me to do a sliding window. That means the 90 day count will include pre-name change question closures. Maybe the chart that Shog provided has something more meaningful.
    – Thomas Owens Mod
    Dec 7, 2016 at 10:12
  • 1
    notice the downward trend in October/November 2016? -- Yeah, and the downward trend from April of last year onward. All of the curves have the same shape, which suggests to me an overall traffic decline.
    – Robert Harvey
    Dec 7, 2016 at 14:19
  • @RobertHarvey The slope is much steeper in October/November 2016. But yes, there are other downward trends.
    – Thomas Owens Mod
    Dec 7, 2016 at 14:20
  • @ThomasOwens, I'm not suggesting a sliding window; I'm suggesting looking at raw counts rather than percentages. The percentages aren't necessarily comparable as the deniminators aren't necessarily the same.
    – D.W.
    Dec 7, 2016 at 16:30
  • 1
    @D.W. I don't have easy access to before and after raw counts. You can maybe get that from SEDE, or take a look at Shog's graph that I edited into the answer.
    – Thomas Owens Mod
    Dec 7, 2016 at 16:31
  • A good-willed answer.
    – samus
    Dec 9, 2016 at 21:01
  • 1
    rather than comparing close reasons I'd rather be comparing close vs number of questions asked. I look at this and it makes me think the site is losing popularity.
    – candied_orange
    Dec 13, 2016 at 5:43
  • of course it is losing popularity @CandiedOrange because until recently site was known as a place where one can post debugging help, tool recommendation, career advice and, as they say "HAVE TIME to read the answer". As this seems to be less and less of an option here, folks who counted on that in the past get disappointed and lose interest, that's indeed decrease of popularity
    – gnat
    Dec 13, 2016 at 20:58
  • @gnat which is all well and good but the point was to increase the signal to noise ratio. Not just kill the signal. This graph only documents categories of noise. I was naively hoping our ratio went up. And that one day we can be popular for good reasons.
    – candied_orange
    Dec 13, 2016 at 22:19
  • fast increase in "signal" seems unrealistic @CandiedOrange. If you think of it, audience interested in site topics has built for over 3 years now, after scope was stabilized at where it is now. It can grow further only organically which is usually slow
    – gnat
    Dec 14, 2016 at 6:06
  • @gnat which is also fine but more likely if the signal to noise ratio actually did improve. Still waiting for numbers on that.
    – candied_orange
    Dec 14, 2016 at 6:09
12

I asked myself the same question during the renaming. I didn't want to make the data we have now fit my interpretation, so a few days after the change I came up with an hypothesis:

Oct 22:

Hypothesis: Over the period of one month, the name change will result in a 20% reduction in the use of the “no debugging” close reason.

Current data for the last 30 days: 517/1241 closed for any reason, of that 177 as debugging. That's 14.26% of all questions. The hypothesis would predict 11.41%. (Data taken from https://softwareengineering.stackexchange.com/tools/question-close-stats?daterange=last30days)

— starting at https://chat.stackexchange.com/transcript/21?m=33068748#33068748, here with fixed typo

Specifically, I'm not interested in the ratio of different close vote reasons to each other, or the total close rate, or the number of questions closed. I am only interested in the percentage of all asked questions that were closed as code/debugging requests.

One month later, I revisited this hypothesis – and it held:

Nov 22:

So one month back, I hypothesized that in the wake of the name change, we would see a 20% reduction in questions closed as debugging.

The numbers are in. Debugging questions have been reduced from 14.26% (177/1241) down to 9.04% (86/951). That means the prediction has held, yay :)

However, total question volume has also sunk by 23%. Are these changes caused by the name change? Or are there simply less off-topic questions due to seasonal reasons? I don't know.

But for this one arbitrarily picked metric, the name change seems to have been a success, or at least not have a negative influence.

— starting at https://chat.stackexchange.com/transcript/message/33659633#33659633, here with fixed typos

(For reference, we are currently at 64/897 = 7.13% debugging questions.)

There are still a lot of off topic questions. But so far, it seems that the name change has helped to get the number of debugging questions down dramatically – my 20% prediction was far too conservative.

The debugging questions are no longer the uncontested #1 problem of this site. Debugging questions are now on par with too broad, unclear, or tool-recommendation questions. Of course every closed question is bad, but they are now on more healthy, more manageable levels. And for that I am thankful.

10

On those things that cannot necessarily be measured with statistics...

The good

  • The number of off-topic questions has decreased significantly.
  • People seem to understand the scope of the site better and refrain from asking questions that clearly don't belong.

The bad

  • People still point to obsolete tags and bad but popular questions from the past as evidence they can ask their marginal questions here.
  • There is still a tendency for people who are site-shopping to ask their off topic questions here when our site is "the closest match."

The ugly

  • Question quality has not increased significantly. If anything, it has decreased, as folks with "name this thing," "which pattern should I use," "review my design," and "am I doing this correctly" questions now ask them with greater fervor than ever before.
1
  • 1
    I've also noticed the "review my design" and "am I doing this right?" questions. I think part of the problem is that these are valid software engineering questions -- in the broadest sense -- just not a good fit for a Q&A site with our goals. I don't know how to fix this :(
    – Andres F.
    Dec 8, 2016 at 3:44
0

I have no data to back it up but I've noticed fewer contensious arguments when a question is closed. I don't think it's that our name is Software Engneering so much as it isn't simply Programmers, a title to which anyone who has ever programmed a speed dial can rightfully lay claim.

Just wish the name change hadn't made Duga so chaty.

8
  • 1
    Part of the reason Duga is more chatty is that she now listens for software in comments on SO. That's an obscene number of comments, including all links and nearly all references to not only this site, but Software Recommendations. Not to mention people just talking about software.
    – Thomas Owens Mod
    Dec 6, 2016 at 1:19
  • @ThomasOwens I know, but can we fix it? A case insensitive match for "Software Engineering" and softwareengineering wouldn't be enough?
    – candied_orange
    Dec 6, 2016 at 1:20
  • 1
    You'd have to ping Simon Forsberg. He's the maintainer of Duga. There's also some Duga-centric chat rooms. I wouldn't mind ignoring some things that contain software, like links to softwarerecs.stackexchange.com. I think going to the Duga chatroom and helping to train would be good, too.
    – Thomas Owens Mod
    Dec 6, 2016 at 1:24
  • @ThomasOwens I go to chat and filter rooms with "Duga" and get no hits.
    – candied_orange
    Dec 6, 2016 at 1:27
  • 2
    Here is where Duga posts everything and responds to training messages. It looks like Simon hasn't been online in a bit.
    – Thomas Owens Mod
    Dec 6, 2016 at 1:31
  • 2
    Apparently the way to train Dura is to reply to her message with "do programmers.classify false" or "do programmers.classify true"
    – candied_orange
    Dec 6, 2016 at 1:57
  • And Apparently SimonForsberg is the guy to bug about tweeking Duga.
    – candied_orange
    Dec 6, 2016 at 2:11
  • 2
    Maybe I should have @Duga scan comments about herself on meta sites as well and have her ping me about them.
    – Simon Forsberg
    Dec 13, 2016 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .