16

One of those ideas that got bantered around - does closing a question have any evidence at modifying a user's future behavior if it happens before they get an answer?

We often see people ask a question, get an answer, and then get the question closed for one reason or another. Also, there are times when the question is closed more promptly before the user gets an answer (I'll ignore answer in comments for this).

So, what I'm curious about if devs can pull the data:

  • If the first question a user asks in the past year (I'm trying to exclude things from long ago) is closed within 48h ...
    • Before an answer is posted to the question
    • After an answer is posted to the question
  • What is the likelihood that
    • The next question remains open for 48h?
    • The next question is closed within 48h?
    • No other questions are asked?

Compare this to the other group of the users who asked questions that were not not closed shortly afterwards:

  • The first question a user asks in the past year remains open for 48h ...
  • What is the likelihood that
    • The next question remains open for 48h?
    • The next question is closed within 48h?
    • No other questions are asked?

What I'm trying got get at here is the "does closing a user's question show any evidence of altering a user's behavior on their next question in a way to provide better content for the site?" It would be gravy to get information about the comments (and if there was a link to meta in any of them), but that really starts going down the data analytics path.

I'll also point out that another dimension to the data is what the question was closed for - unclear, too broad, primarily opinion, duplicate, off topic (reason vs custom), and migrated. Again, that might be going too far down the data analytics path. If there is data for this provided, I suspect one would see unclear still likely to ask another unclear question - the other options though, I wouldn't bet on going one way or another.

1
  • 3
    I am trying to put a query together in data.stackexchange.com. I feel bored and like a challenge
    – maple_shaft Mod
    Commented Apr 29, 2015 at 1:21

1 Answer 1

Reset to default
15

I wrote a query that provides the raw data for the answer. You can find it here.

http://data.stackexchange.com/programmers/query/307939/first-posts-closed-compared-to-second-posts-closed

The raw data shows first posts by new users within the last year from today where there first post was closed within 48 hours. Each post in this subquery is ranked so that two additional subqueries can be created to return just the post data where rank is 1 and 2 respectively, their first and second post.

The first and second post subqueries are joined to the Users. The first post obviously being an inner join but the second post subquery being left joined so we can see which users had their first post closed within 48 hours only to never return.

EDIT:

A profound thank you to Air for finding a problem with my query. The original had a big flaw in that the SecondPost view was actually ranking all of a users closed questions! This was clearly wrong, so I updated it and the data gives a much less alarming story now.

https://docs.google.com/spreadsheets/d/1Bb1a4Dd8KM-BksS4EmgKzlJG7BstkLp8pi-cye4pqlY/edit?usp=sharing

This spreadsheet basically describes a less alarming situation now that we have that fixed. Here is the rundown:

Total users with first question closed in 48 hrs in last year: 591
Users from above with a second question: 76
Users whose second question was closed: 29
Users whose second question was closed within 48 hours: 26
Users whose second question was closed after 48 hours: 3

Percentage of users that returned with a second question: 12.86%
Percentage of users whose second question was closed: 4.91%
Percentage second question closed within 48 hours: 4.40%
Percentage second question closed after 48 hours: 0.51%

This actually shows a much more optimistic picture now. I would have guessed that probably 4 out of 5 closed questions on programmers from first time users are from people who even if they understood the sites scope, it would be of no interest or concern to them. Eg. PlzTehCoders, People who really want opinionated Q&A like Quora, Ban Circumventers, etc.. The data supports my initial thoughts.

The good news is that the majority of the people that post back have a question that is now well received by the community! Secondly poor quality posts are mostly getting resolved within 48 hours. What this means is that we are not only doing a good job of policing content, but we are doing a good job of coaching new users into producing quality content on their second try.

Good job Community!

13
  • it would be helpful to run these queries against data that includes deleted posts (it's not available in public data dumps)
    – gnat
    Commented Apr 29, 2015 at 9:34
  • @gnat I suspect the data would be similar. To a new user nay an event be more devastating than having your first question to a new community be rejected. It is a strike against the ego and we naturally try to avoid such social stigmas. Basically I think there is little difference to how new users perceive a closure vs a deletion. Furthermore it would be more complicated to query because we would want to rule out self deletions.
    – maple_shaft Mod
    Commented Apr 29, 2015 at 10:53
  • agree, it's probably so. It would be helpful to have more accurate stats anyway, if only to confirm that it's indeed so. As for self-deletions, that's an interesting turn. Would be interesting to learn more about them. I'd guess these folks tend to be more conscious and more likely to learn because of that, but that's only a guess
    – gnat
    Commented Apr 29, 2015 at 12:07
  • 3
    @maple_shaft consider that the questions that are closed without answers are likely automatically deleted by the Roomba script. If the first question is such, one wouldn't see that in Data Explorer and instead see the second question which wasn't closed. It would indeed be helpful in comparing the data... and thank you for building the tools for digging into such questions as best we can.
    – user40980
    Commented Apr 29, 2015 at 15:36
  • @MichaelT That is a good point. I forgot about the cleanup script.
    – maple_shaft Mod
    Commented Apr 29, 2015 at 15:57
  • 2
    The RANK() function in the first CTE is not performing as intended - it's ignoring open questions when determining a user's Nth question, so actually giving the user's Nth closed question. This is because the ranking function is applied after WHERE filters the result set; see: select a, rank() over (order by a asc) from (select 1 a union select 2 union select 3) t where a <> 2; (this is probably where the 100% is coming from)
    – Air
    Commented Apr 29, 2015 at 16:57
  • I'm surprised at the surprise. If you'd just joined some new website, and the very first action you took was officially slapped down by 5 established users, wouldn't you be irritated?
    – Michael Shaw
    Commented Apr 29, 2015 at 17:52
  • @MichaelShaw from my days in usenet and mailing lists - nearly every new user who didn't read the faq had similar experiences, and yet they came back and some reasonable percentage became useful contributors. The ones that were corrected as to how to participate in the community, yes, they suffered a blow to the ego (myself included in the day), but it was an important part of trying to figure out what is acceptable. The question remains of "What effect are our corrections (given the tools we have) having on future new user behavior?" I suspect some leave, I know many stay.
    – user40980
    Commented Apr 29, 2015 at 21:11
  • @Air My goodness... thanks for clearing that up! See my edit, I got a very different more optimistic picture :)
    – maple_shaft Mod
    Commented Apr 30, 2015 at 2:02
  • 1
    @MichaelT Please see my edits. The query had a problem, and now that I fixed it there is a much more optimistic picture now!
    – maple_shaft Mod
    Commented Apr 30, 2015 at 2:03
  • 4
    Results of same query, including deleted
    – Shog9
    Commented Apr 30, 2015 at 2:41
  • 1
    It would be interesting to run this query on other sites - including SO to see whether we're "normal" or not.
    – ChrisF Mod
    Commented Apr 30, 2015 at 8:45
  • Eh, when were we normal at anything @ChrisF? ;)
    – yannis Mod
    Commented May 8, 2015 at 11:20

You must log in to answer this question.