Shared posts

16 May 14:48

quick lit – rapid evidence reviewing

by pat thomson

This is one of a very occasional set of posts about some of my own academic work that you might find useful.

A colleague and I have just undertaken what is called in the (academic) trade a Rapid Evidence Review. Or, as I have come to think of it, Quick Lit.

An RER is a form of literature review which is popular with policymakers and with organisations seeking to design and/or commission research. It aims to establish what research is available about a defined topic, as well as key results and major gaps. RERs are usually done by academics who already know a field of research well, as was the case with my colleague and me. When researchers bring existing knowledge of the field to an RER it makes for speedier work.

In this post I’m going to describe our RER process.  I’ll use the term evidence throughout, even though you and I know that this is a highly contentious notion. It is however the terminology that is used when you are doing this type of literature work.

My goal in describing the RER is simply to make explicit one strategy for reviewing literature. It’s a strategy that you might want to use – and adapt – if you have to do a roughly similar task.

IMG_0070.JPG

Steps in the RER (Gough, Oliver and Thomas, 2013 p. 11)

Another message to take from this post is that there is not one way to do literatures work. The literatures strategy you use often depends on the kind of literature review you are doing, what you hope to get from it, and who you are doing it for.

Beginning a rapid review – what’s in and what’s out

The RER begins with one or more focus questions. You may have to define these yourself – in our case, these were already established. Our RER commissioner had six questions that they wanted us to address.

Next, the boundaries of the RER have to be set. In our case, these were negotiated with the commissioner. The decision is about what literatures will be included and excluded and what are the cut off dates. In our case, we were interested in UK literature only, literature published since 2007.

The search terms – key words using for searching – and sources – the data bases searched – are then determined. And in our case, the terms and sources were agreed with our RER commissioner.

Sorting out categories

The difference between the usual form of systematic review and a rapid evidence review is that a systematic review often excludes research of particular types. Because we had to ascertain what kinds of evidence there were on our topic, we had to take an inclusive approach –  our task included categorising research types.

We determined a numerical coding system for different types of research evidence.  We used a fairly standard system for categorising research approaches

1 a. systematic review b. meta-analysis

2 randomised control trials

3 a. longitudinal studies b panel series c cohort studies d. secondary data analysis e. other

4 case-control studies. b. case series c. case reports d. mixed methods e. survey f. interview-based study g. ethnographic study h. theoretical development i. other

5 expert opinion

6 any other (eg thesis)

While this list could be – and sometimes is – read as a hierarchy of research types, we and our commissioners were clear that different research questions often require different approaches.

With this list in hand we then used the two number classifications – research question, and type of research –  to construct a table. Our table also had four other columns. Six in total. See below. The first column was the bibliographic information to be used in referencing. The second was the country in which the research was conducted (where the data was from). The UK consists of four nations and it was of interest to see which countries the research addressed. The fifth column was further details about the research method – its sample, size scope etc. The sixth and largest column was for the key results. Anything we wrote in this column had to be short and pithy so that we could do the required task of identifying key results, debates and gaps.

IMG_0071.JPG

I must pause here for a caveat.  Our review was designed to meet our commissioners’ needs, but the codings can of course be varied. Different types of search might have different columns. At another time and in a different review we might, for example, look at the gender and race of the researchers. A further column could be added to look at key definitional terms. Or theoretical resources. The point is that using tables and coding categories allows you to do some counting and comparing. This may be useful.

Categorising and noting

It took us a few days to complete our first wave of searching. And when we had all of the relevant papers in hand, we went through them systematically, firstly recording their bibliographic information and country. Note, we didn’t search and note, search and note, search and note. We did one big search and then noted the corpus.

We didn’t sort our list of papers, reports and books alphabetically, but we could have, although in our case it wouldn’t have made things much easier. I would also advise using bibliographic software for this section of the task, importing papers and reports as PDFs with associated bibliographic information into a project library. Only some books are likely to need to be entered manually. But the software entries do need to be checked for accuracy.

We read the methods section of each paper in order to categorise the type of research used (as above, types 1-6). We then read as much of each paper as we had to, to ascertain which of the six pre-set research questions an individual item addressed. And then we sorted and summarised the key messages. Sometimes this meant we read the abstract, introduction and discussion/conclusion, often more.

Once we had all this information we were able to fill in the table we had constructed. Item by item.

We used a word document for this task, but we could alternatively have used an excel spreadsheet. However, as most of the items we found addressed more than one of our six key research questions, we would still have ended up doing a lot of counting by hand.

Looking for search omissions

We had to make sure that our initial search had located all of the literatures. So we added a few more terms just in case. In our RER we had two more waves of searching; the first to catch any recent literatures we had missed, and the second to pick up some key international literatures of research types 1-3. We looked outside the UK because we suspected, on the basis of our first wave search, that the field we were investigating had very little of type 1-3 type research at all, not just in the UK.

Sorting the master list

We eventually had a master list of publications. Every item completely categorised. We were then able to sort this master list into six sublists – each one addressed one of our specific six questions. These six lists then became the basis for a very critical evaluation – much discussion between us – and this led to a written summary of the literatures.

IMG_0069.JPG

Writing the report

Our written report began with a description of our approach to, and process of, undertaking the RER. We then described the overall body of evidence, giving numbers and types of research for each of the six questions. A summary of key results for each of the six questions was provided. We concluded with a description of the gaps in the research and strategic possibilities for further inquiry.

Our RER, done.

The RER approach forces you to be quick and succinct. It is obviously not a process you want to use if you are seeking to get deep into the literatures. Even the usual systematic review takes longer than an RER. However the RER approach can be adapted – for example, for an initial scoping exercise of a field.

Maybe this is a process you could use. But a little reminder, mainly for my sake, not yours.

The Rapid Evidence Review is not a process suitable for all literature reviewing, although there is a family resemblance between all types of literatures work. Pretty well all lit reviews aim to sort, classify and summarise patterns. Questions and tables are often key to how this work gets done. However, a quick lit approach may not be what you want or need. But it could be helpful to you at particular times and for very particular tasks.

Reference

Gough, D., Oliver, S., & Thomas, J. (2013). Learning from research: Systematic reviews for informing policy decisions: A quick guide. The Alliance for Useful Evidence, 1-38.

The entire website in which this OA report is located was down as I revised this post so I have temporarily linked to my copy  Alliance-FUE-reviews-booklet-3

11 Sep 11:17

avoiding the laundry list literature review

by pat thomson

I’ve been asked to say more about the laundry list literature review. The laundry list is often called ‘He said, she said” – as one of the most usual forms of the laundry list is when most sentences start with a name. And the laundry list is a problem. It’s hard to read and not very fit for purpose. 

So, what does a laundry list look like? Below is a page of a published book. It is taken from a chapter reviewing the literatures on neoliberalism in ‘the university’. It’s a laundry list. I have:

  • underlined in red the sentence where the author says what they are trying to do (you might call this a topic sentence)
  • circled the sentences that feature a scholar as the subject of the sentence.

IMG_1647
Now let’s see what’s going on in the writing. The second paragraph on the first page begins with the author’s intention – to establish that it is difficult to define neoliberalism. They then offer two supporting pieces of evidence for this contention – one from Ferguson and the other from Bell and Green. But is the point of this paragraph to say who said what? Well no. It’s the various interpretations of the term that matter, not who was responsible for them.

Let me re-draft these two sentences to focus on the author’s intention.

Clarifying the concept of neoliberalism is not an easy task. The term neoliberalism is often used as a synonym for capitalism or the inequalities of the economy more generally (Ferguson 2010). Some scholars use the term very loosely, drawing connections between unrelated life events to suggest that a clandestine power is ‘pulling the strings’ (Bell and Green, 2016, after Latour, 2005).

What is now made clearer in the re-draft is that this is actually not yet a paragraph. I’ll continue by adding in further information from the next two paragraphs.

Clarifying the concept of neoliberalism is not an easy task. The term neoliberalism is often used as a synonym for capitalism or the inequalities of the economy more generally (Ferguson 2010). Some scholars use the term very loosely, drawing connections between unrelated life events to suggest that a clandestine power is ‘pulling the strings’ (Bell and Green, 2016, after Latour, 2005). Neoliberalism is also almost always derogatory when used to refer to economic/political policy (Fish, 2009) which produces austerity through the rationality of markets, entrepreneurialism and competition (De Lissavoy, 2014).  The term is also associated with ‘bureaucratisation’ (Hibou, 2015), processes of rationalisation and professionalization, driven by the quest for neutrality, objectivity and professionalization, which govern key aspects of everyday life.  

So that will do as a new draft. It can be worked on further later.

The next paragraph should move on from the need for clarity to offer the definition that the author will use.  Now let’s look at the next page.

IMG_1648Let me try a bit of a rework on that too.

How then can the term neoliberalism be understood? Barnett (2005) suggests that it refers to the discreet alteration of the class-driven reform of the state to benefit free markets. Neoliberalism is a form of ideologically driven policies and government that supports privatisation, the free market and increased competition.

 You can see that I have left Barnett here as a sentence subject. It is not that you never write about an author. The reason I have left Barnett here as the sentence subject is because he is The Key Scholar that the author uses for the definition that informs their book. While we might not agree with them or their definition, when we read Barnett in the sentence, we are clear on what authority the author’s work rests. ( The same is also true for scholar Springer at the top of page one. Springer is also a key source for the author.)

And note, my new second paragraph splits the author’s current third paragraph. I’ve turned three non-paragraphs into two. But perhaps they need some further evidencing/referencing. Yes, it is pretty obvious that my new second paragraph in particular needs a bit more work –  more evidence and argument would strengthen the case being made as well as showing the breadth and depth of reading in the field. You see, once you get away from the he said she said list, you get to show that there is a quantum of evidence for the point you want to make, rather than a less than persuasive single citation. (And note that in re-drafting I have got rid of the ‘therefore’ sentence – this is where the author has tried to reinsert their own post-listing voice and interpretation.)

In sum, my redrafting has:

  • collapsed three paragraphs into two, each one makes their own move in the argument. The first point is that there is confusion about the term neoliberalism, the second offers a working definition of it.
  • avoided the repetitive use of he says, notes, proposes etc.
  • changed some klutzy expression – I moved into brackets the cumbersome double reference – Bell and Green drawing on Latour – so it’s now not too tricky to read.
  • made the writing more authoritative – I removed the abrupt shift into the author’s own view via ‘therefore’ –  the entire two paragraphs are now the author managing the discussion of the substantive topic.
  • moved some sentences from passive to active voice – see paragraph two in particular.
  • highlighted the most important work that the author is using – Barnett
  • produced an argument – I’m not simply reporting summaries of other people’s work, but have made two points supported by evidence.

And the two paragraphs are now ready for further polishing.

If you go on reading the second page, you’ll see the listing pattern repeated. The author’s next paragraph is about how neoliberalism has permeated the university, as is the one after. I’m afraid the first paragraph on page two doesn’t even have an opening sentence about the topic – instead there is a give away sentence about writers. And then comes the list and the author inserting themselves at the end trying to make their point.

The same process of de-listing and re-writing that I’ve already done could be done here. You might even like to try rewriting this text yourself, particularly if you are still working on how to move away from listing.

But the original wasn’t a disaster. It’s not that the author doesn’t know what they want to say. They do. They have grouped the relevant literatures into clumps that move through a tacit argument. And they do have a point to make. It’s really that the text could have been much better.  

But that’s the laundry list issue for you. By focusing on writers rather than the substantive issue under discussion, authors end up listing and then trying to draw things together. They write paragraphs that aren’t really paragraphs but collections of sentences lacking a sensible opener and closer. Laundry list writers don’t really manage the argument – their case is nowhere near as clear as it ought to be. And when this pattern of listing goes on and on – as this one does for an entire chapter – it becomes a repetitive and dull read. In this particular book, this is the only chapter that is so dreary; the rest is much more readable.

It is no accident that it is the literature chapter where such problems occur. Listing is often an issue in thesis literatures work. If you currently have a laundry list in your literature review, then see it as a draft. It’s not too late or hard to rewrite it – you just need to take charge of the text. 

Don’t let other authors hog the lime-light. It’s your work and you need to tell it how it is for you.


26 Jun 23:22

Refresh Types

The hardest refresh requires both a Mac keyboard and a Windows keyboard as a security measure, like how missile launch systems require two keys to be turned at once.
26 Jun 23:20

three things examiners look for in methods chapters

by pat thomson

Once upon a time, when I worked in schools, early childhood teachers routinely issued young children with a ‘pen license’. A pen license was much sought after as it meant that a child could ‘advance’ to using a pen instead of a pencil. Using indelible ink meant that the child was able to write legibly in longhand. But legibility wasn’t enough, the child also had to be able to copy and compose text without making lots of mistakes that needed to be erased. Writing in pen meant the pupil had been deemed competent at basic writing tasks.

Of course, while schools issued rubrics about what counted as the standard for the pen license, different teachers did interpret the rules slightly differently. And different children learnt differently, so they didn’t all achieve the license at the same time. However, by and large, it seemed that most children got their pen license well before going on to ‘big school’ in Year 3.

Why am I telling you this? Well, the modern PhD works a bit like a pen license – the PhD is a license to practice research. Once you have the PhD you can apply for grants in your own right, and you can also teach other people about research.  And you can supervise doctoral researchers. … although contemporary universities and funding bodies often require newly minted PhDs to keep working with a more experienced researcher in all of these tasks. But in theory you know – you have your research license…

Now it’s not unhelpful to think about the thesis examiner as responsible for granting a license to research. Understanding this helps you to focus on the things that the examiner looks for as ‘evidence’ that they can licence you … 

A key place where examiners focus their licensing attention is on the chapter which outlines the design and conduct of the research. The methods chapter, as it is often called and as it appears in the thesis, is not a mindless ritual, something to be done just because it is the done thing. It is actually the place where you demonstrate that you know how to design and conduct research – not just this one project, but others in future. You have to show you understand and can use the principles of good research.

Yes… what counts as good research design and research conduct does vary from discipline to discipline and also by research tradition. So it’s good to start your thinking about licensing by noting that your examiners will have been selected because they are expert in the discipline and tradition you are using, and have been institutionally deemed capable of deciding whether you will be deemed capable and competent.

However, despite all disciplinary variations, there are three things that most examiners look for in a methods chapter. They look at 1. research design 2. data generation and 3. analysis.

8981026019_f3b215be94_k.jpg

So I’m going to look at each of these questions in turn, but a health warning. Do bear in mind that what I’m going to say is pan-disciplinary and across research traditions and isn’t comprehensive. You still need to think about what these questions mean for you and your particular research.

  1. Is the research well designed?

Examiners look at the nature of the research question – is it answerable? Have key terms been explained in terms of this study? Are there any ambiguities in question wording that the researcher hasn’t taken account of?

The examiner will consider whether the methods chosen are fit for purpose – this whether they will produce data* that will allow the question to be answered in some way. Your choice of what or who is involved in the research (sometimes called the sample) are vital. Examiners will make a judgement about whether the data to be generated by this design will be sufficient  – will it be ‘good enough’. In other words, they will anticipate the nature, quality and quantity of data that your methods will produce. And examiners will pretty well always look at the proposed analytic approach to see what kinds of understandings it might produce – and how this relates to the ambitions expressed in the question. Examiners  also look at the potential for difficulties to arise in the research – these might be ethical or analytic. Examiners look not only for fit but to see that you understand what this design can and can’t do.

2. Are there any problems in what the researcher actually did to generate data*?

 Examiners are looking for processes where there aren’t blank spots – some things are where  the researcher ought to have seen as a problem but hasn’t. Inevitably the research will do some things and not others, and examiners want to know know that you know this. So they make sure for instance that survey questions have been well constructed, experiments are carefully designed, that interview questions aren‘t inappropriately restrictive and closed, that observations have been focused in line with the research question … Examiners look for consistency of approach, regardless of whether the research is an emergent process, highly structured or something in between. They are also looking for how the researcher puts their ethical undertakings into practice.

3. Has the data* been analysed thoroughly?

Now I’m going to devote another post to this last question, so I’ll only say here that examiners look to see that you have used a rigorous process which has been consistently applied. This might be a statistical approach which involves complex modelling – examiners look for example for definitional assumptions, choice of technique, accuracy in application and calculations, appropriate tests for significance and so on.  Or it might be a contemporary ethnographic approach which involves writing processes, from field notes through to descriptions and then theorisations – the examiner looks here for the explicit chain of conceptual development. Or it might be the more usual qualitative approach of building themes and coding – the examiner looks for the processes used in transcription, the ways in which inductive and deductive connections have been produced, the development of categorisations and ‘coverage’ of the raw data and so on. Or it might be a process of interpretation which connects together various kinds of primary sources – the examiner looks for what sources, how they have been used, omissions and un/tenable assumptions…

The examiner wants to know whether all of the data that was generated has been used, whether disconfirming data was taken account of and how. They also want to know how any conceptual framework and/or theorisation was developed as part of the analysis.

Then… The sum of these three questions leads the examiner to decide whether the researcher gets their license. In some research traditions, the examiner is understood to be looking for research which is both valid and reliable, perhaps also replicable – in other traditions the examiner looks for the trustworthiness of the research process and the credibility of interpretation.

So what does knowing the importance of the three questions mean for the doctoral researcher?

Well, it means that the thesis, and particularly the chapter outlining the research process, has to provide the examiner with sufficient detail to make judgments. Examiners want to see:

  • An explanation about why the research was designed in the way it was. Why this research question is worded in just this way and then the what, why, who, when, how many, how often…
  • A discussion about the ways in which you generated data, including any problems you encountered. Some of the more detailed points as well as samples of the actual data ‘types’ may be included as supplementary material.
  • An audit trail. A lot of this material can be confined to appendices. ( Note that not all examiners like appendices. Check out their preferences if you can. Me, myself and I love a good appendix.) The examiner usually requires  a list of all of the data, both raw and treated and relevant details such as place, time, source etc. They mostly want to see samples of the analysis –in some research traditions the entire data base and all of the workings are provided for checking, while in others selected anonymised data are presented. Documentation of the decisions made during the research – about data generation and its processing – provide a map for the examiner which shows how the project was brought to the point where it was written as a thesis.

So this all means paying careful attention to the places in the thesis where you present your examiner with the information about your research.

What do examiners not want to see? Yes, this is a serious question. With an answer. 

Don’t give the examiner a text without sufficient methods information. Nor the reverse. You don’t have to bore the pants off the examiner with mind-numbing detail, so do discuss with your supervisor what examiners need access to immediately, and what they need to access in appendices or supplementary materials in order to see the complexity of your work.

And remember…thinking about these three questions means that you’re well on the way to becoming licensed. 

A caveat

*Data is a tricky term. I’ve starred it just so I can say that it is usually applied to numbers, images and words, and all combinations of them. Data can be generated from texts through work in libraries, through numbers generated in labs or in selected sites such as offices, schools, factories and hospitals.  But there is also debate about whether there can be any such thing as data, and post qualitative researchers now seek other ways to conceptualise research practice altogether. This is a blog post so I can’t do much more than just show that I know that this term – like just about every other one I’ve used above – are contentious!

Image credit: Rich Remoneron


17 Oct 09:20

95% dos clientes recomendam a ArtVision

by carolinacvcustodio@gmail.com (carolina)

A ArtVision realiza anualmente um inquérito com o objetivo de avaliar a satisfação dos seus clientes. Dos vários pontos em análise, viagra 40mg é colocada uma questão com resposta directa: Recomendaria a ArtVision a outras entidades?

Trata-se de uma questão muito importante, pois a recomendação, pressupõe confiança, e neste ponto a ArtVision contou com resposta positiva de 95% dos seus clientes.

Este resultado deriva da opinião dos clientes que destacam na equipa ArtVision a competência e sentido de responsabilidade como pontos fortes.

11 Sep 10:29

Hypothes.is plugin for OJS

by Alec Smecher
I was fortunate recently to attend a conference at Schloss Dagstuhl in Germany called
21 Oct 11:14

Voir Jewellery Stand by Ana Tevsic

by Harry

Wood cones and marble depressions stand in for deep forests and cold rivers in Ana Tevsic's Voir Jewellery Stand.
01 Apr 10:51

Introducing Gmail Blue

by The Gmail Team
Posted by Jonathan Zames, Project Manager

Gmail launched nine years ago on April 1st, 2004. Since then you've been able to use hundreds of new features that push the boundaries of what email can do and make it easier to get things done.

Starting today, you'll get to experience the next big step for Gmail, Gmail Blue. Watch the video to learn more:


Update: Was Gmail Blue an April Fool's joke? Take a look at developer Shon Dalezman's Gmail Blue Chrome Extension and then decide.