I’ve been using SciRate for 2 1/2 years. I began using it with certain expectations, but my actual use has differed from those expectations.
The simplest and most used feature of SciRate is its “Scites” button. With one click, SciRate members can vote for a paper. Initially I wasn’t sure how I should use this feature. What did my vote mean? Should I only vote for a paper I had read? Did it mean I could vouch for its correctness? Eventually my selfishness kicked in. SciRate made it easy for me to see what papers I had scited. And sciting a paper was so light-weight that it became the easiest way for me to mark papers that I wanted to come back to later. I don’t always come back to those papers, but I frequently use my list on SciRate to find a paper whose abstract I vaguely remember reading, or to find a set of papers it would be fun to read over the weekend.
I’ve followed arXiv postings for over a decade. Sometimes I skim new postings every couple of days. Other times I’ve gone for weeks without checking. SciRate, by enabling me to filter my review to only papers that others have scited, has given me a wonderful way to catch up after a period of disuse.
The most unexpected positive result from SciRate was an invitation to collaborate. That use now seems obvious, but it was a surprise when it happened. A fellow researcher contacted me about the possibility of collaborating on a project he believed I would find interesting. I was interested in the project he described, but I couldn’t figure out how he knew that. None of my papers were in that area. All I had done was read about it. How did he know? Then I remembered that I had scited papers in this field. In particular, I had scited one of his papers the previous week. Sure enough, that is what prompted him to contacted me.
While I have scited over 90 papers, and use SciRate frequently, I have yet to comment on a paper. Why haven’t I? I don’t know the full answer, but there is one type of situation in which I thought I would comment, but I was wrong. I thought I would post comments on papers I had reviewed. I figured I could just post the review, or part of the review, as is. But reviews have a different style from comments. Posting suggested improvements did not seem valuable to the community at large, and I knew the authors would receive my comments via the normal channels. Commenting that I considered a paper to be a “weak accept” seemed inappropriate. In short, even commenting on papers I had reviewed required effort, and the pay-off wasn’t clear. I still intend to use SciRate to comment on papers. It will be interesting to see what will prompt me to do so.
It seems that the real advantage of SciRate is not so much as a vehicle for criticism or dialogue, but as a way to determine interest and to network. Interacting with a publication can be made easier by creating new communication methods and reducing feedback to “point-and-click”, at least in theory.
For my part, making substantive comments about someone else’s paper is not something that comes easily because I want my comments to convey the impression that I’ve actually read the paper and thought about it. That requires time that I might not have unless the topic is something I am really, really interested in. As Eleanor points out, there are other channels that already exist for those kinds of communications. I also find myself wondering if my comment is going to state something obvious to the author, but did not find its way into the finished paper for reasons unknown to me.
The trick of using the system to track what others find interesting way to help deal with the crush of data from the ongoing avalanche of papers.
People are reluctant to write comments for the various reasons that Eleanor and you point out because the benefit to them of writing the commentary is low. If, however, there were cultural or societal incentives to write reviews (for example, if tenure committees expected it) then the investment of effort would have a predictable return, just as the investment of effort to write a paper has the promise of publication. If the effort is seen as valuable, it should be possible to build up a reputation economy around the effort.
Gene – Yes, incentives are an interesting issue, not just for this question, but also for conventional paper reviews, and scientific publishing more generally. Setting up appropriate and effective incentive structures is non-trivial however. (Tenure committees expecting comments would have little effect on me! :) )
I should perhaps have said in my original post that I rarely read comments on SciRate. Just now, I tried to remember if I’d ever read a comment there I found helpful, and couldn’t think of an instance (as opposed to the scite total, which I use frequently). That seemed surprising, in part because I knew about SciRate through the blog of Dave Bacon, who runs SciRate, and I had found his blog precisely because he had blogged about a paper I was interested in. Since then I’ve followed Dave’s blog. So today I wondered why I don’t follow his comments on SciRate. Turns out he has only commented on 9 papers, with four comments on one paper for a total of 13 comments. On the other hand, I found one of these comments interesting, so I should have done this search before! Anyway, I suspect that not having found other people’s comments helpful contributed to my lack of motivation to comment myself. Paradoxically, a high standard of comments might actually increase motivation to comment even though it makes writing comments in the first place more challenging.
P.S. Here’s the story of how I first came across Dave’s blog. Nature published a paper “Quantum Information can be Negative” in 2005. But I had read a paper many years before with this result so was puzzled as to what was going on. Unfortunately I couldn’t remember the authors or title of the paper so I was reduced to a web search. The first hit was Dave’s blog post which not only had links to the 1995 paper I remembered, but also answered my question by explaining the significance of the newer paper.