How to evaluate sources

There are five important pieces of information you can use to evaluate a source you find via web search or cited in another source, to decide the priority of study and the depth of study. The two most informative are:

Other key points are

Note that I emphasized informative. That's not the same thing as important. The importance of the factors varies case-by-base. If you didn't read the abstract, it can't be important!

Priority of study

There are two decisions to make: what to study next, and how much to study each resource. These decisions are made in very different contexts! To make a good decision about what to study next, you need to evaluate each of the list of resources you have.

There are many sources. Frequently Google Scholar will tell you there are millions of academic sources that fit your keywords. And of course there are likely millions more non-academic sources that may shed light on your research, from newspaper articles on social implications or business launches to commercial "white papers" and case studies. You need to decide what to look at first.

You can try to improve your keywords, but even so you're likely to get hundreds. If you make them too specific, you will get many fewer but you will likely miss many important ones. So you will need to make decisions. Also, you don't know exactly what Scholar's algorithm is. It often makes good decisions about what to offer first, but usually most of the top results aren't very good.

In a Scholar search, the first thing to look at is the quoted text (not the title!). If your keywords in that text don't "fit together" as you expect, or the content of the text doesn't make sense for your research, you can throw that one away. (Maybe look at the title, first.) If the content seems to be useful, then look at the title. If that seems related to your project, it's high priority. If not, it's low priority. If you know any of the authors, that increases priority. Record both high and low priority items in your resource list.

Of course you can't possibly check hundreds, let alone millions, of results in a reasonable amount of time. So you will just have to trust the search algorithm to bring up better results first. Once you've got "enough" high priority resources, stop, and start accessing them. Ten is a good estimate of "enough" to get started, but sometimes you go through a hundred or more Scholar results and find only a few "high priority" sources. You may continue a bit, but if less than 1 in 10 of the results seems interesting, it's probably time to take another approach.

There are three ways to renew your search. The first is to change your keywords and search Google Scholar again, or try a different search engine (regular Google, Bing, Yahoo!, DuckDuckGo). The second is to take the sources you have, and "mine" their bibliographies for new sources. A bibliography which is very poor in good sources is a strong indicator that the paper itself is unreliable. (I wrote "good" rather than "new" because as you become familiar with your field of research you will encounter the same resources repeatedly as citations in the better sources.) A source that is cited repeatedly in different papers is usually an important source, of course. (The exception is when the citing papers are all by the same author. This may or may not count as multiple cites depending on how different the citing papers are from each other.) The third is to use a citation index. A citation index is a reference work where you can look up a particular source, and find a list of newer sources that cite it. This also works with Google Scholar and other search engines by putting in the title in quotes.

Depth of study

You also need to decide how much effort to put into each source. To some extent you decide this as you go along in a particular source, but even reading and understanding an abstract can take many minutes if you are not familiar with the topic of the source. So how do you decide to read the abstract, or having read the abstract, to read some of an article?

You do have to read all the abstracts of your high-priority sources. The trick is that if you don't understand one because the terminology or the concepts are unfamiliar, you can put that one back on your to-do list with medium priority, and go on to the next one on your list. Once you've learned more about the field, you can come back to it. If you find this is happening repeatedly for the same reasons, then you need to pick one and study it more carefully. You can look up the terms in dictionaries (especially specialized dictionaries for your field of research), or on the web. You may need to acquire a textbook.

The considerations for studying a source in depth are similar.

Content of abstract

The abstract is usually the most important factor in deciding to study a source. In the abstract you are looking for the relationship to your research theme, of course. But that's probably evident already since you have gotten to the point of reading the abstract.

Writing abstracts is very difficult, because you usually have very limited space (sometimes as short as 100 English words or 250 Japanese characters). So a good abstract is a strong signal that the author is a very good writer. That doesn't always mean they are a good researcher, but at least the source will be easy to read!

Quality of author

Content of title

Quality of publisher

Here is a short discussion of different ways to calculate impact factors in Japanese.

Quality of citation

https://www.researchgate.net/publication/342623066_Journal_Citation_Reports_2020_Impact_factor_Ranking_of_2019

https://www.scimagojr.com/