Exploring Hardship: a social media listening explainer
This article by Sophia Knight, Researcher at Demos, is a guest blog for the Joseph Rowntree Foundation, in a series explaining how social media listening could be used by researchers to understand poverty, as well as how charities and other third sector organisations can use these findings in their services and how MPs, policy researchers and decision makers can use social media listening to inform policy. The programme also features a report, which you can read here.
In Summer 2023, JRF’s innovative new Insight Infrastructure team commissioned CASM, the digital research unit at the cross-party think tank Demos, to complete an exploratory project examining how people with lived experiences of financial hardship are talking about their experiences online.
Throughout this series of blog posts, Aleks Collingwood has detailed the reasoning behind JRF’s decision to commission this project and highlighted the key findings, as well as given tangible guidance into how these insights can be used by MPs, charities and other third sector organisations. Her most recent blog asks potential users of these insights about the best format to publish what we find in 2024 to ensure maximum engagement.
In this final guest blog post, I will be giving an overview of how JRF and Demos collaborated to develop and apply this methodology, with a view to encouraging others to do the same.
1. What did we do?
Building on CASM’s extensive experience in social media analysis, we designed this exploratory methodology to examine how people are experiencing hardship day-to-day, in real time, and responding to social, economic or policy changes. In these open, often anonymous forums, we hypothesised that people would feel more able to share their feelings and experiences. As a result, there is a huge amount of information and understanding that can be gleaned from what people are saying in these spaces.
2. How did we do it?
Our methodology was developed through an iterative and exploratory process. Our initial selection of three forums for data collection was informed by expert input from both JRF and the Grassroots Poverty Action Group, a diverse group of individuals from across the UK with direct experience of poverty. In order to protect users’ privacy, we refer to these forums in our report as ‘GovernmentHelp’, ‘FamilyHelp’ and ‘FinancialHelp’. We collected data from these forums through a combination of official platform APIs and web scraping.
Once the data was collected, we began each stage of analysis with an open exploration, starting with searching the dataset for the keywords we gathered from our discussions with JRF and GPAG. We tested our starting assumptions for which topics would be relevant against the data, and refined our queries in response to our initial findings.
First, we undertook a broad theme analysis, using ‘clustering’, a Natural Language Processing (NLP) technique. This technique helped us to discover characteristic terms, phrases and discussions occurring within a large dataset, which we then interpreted to investigate key themes arising. The advantage of this is it allows themes to emerge from the data which might not have been expected to be present, and means that new experiences, challenges and solutions can be surfaced.
Then, within these broad themes, we explored our dataset in-depth through close qualitative analysis. Qualitative analysis allows for greater analysis of the complexities’ of individual users’ experiences, digging into what these themes actually mean in practice and drawing out the details of people’s experiences.
Combining the application of NLP techniques to identify key avenues of exploration enables a more systematic approach to be taken to close qualitative analysis of social media content, and for unexpected themes to emerge, but generalises over a highly diverse dataset. The qualitative aspects of our research ensure that we are still drawing on policy expertise and lived experience. A mixed methods approach, as we have taken, enables the benefits of both of these approaches to be combined.
3. What does that look like in practice?
One of the most powerful insights from our research was seeing, in people’s own words, the impact that financial hardship has on family, friendships and romantic relationships, and how relationship breakdown is exacerbated by financial hardship.
This importance of relationships was initially identified in our topic model, as themes related to child support and family separation were seen not just where we may have expected, in our FamilyHelp data, but across all three forums. We used this high level theme of family and relationships to inform the iterative cycle of building our keywords list, starting with a simple list of terms (including kids, mum, dad, and so on).
Using exploratory methods such as ‘word embeddings’ (see p. 47 of full report), we found additional terms which frequently appeared alongside our initial keywords, and used these to expand our list. Once we had our final keyword list, we filtered our total dataset to just posts and comments containing these terms, creating a new dataset for in-depth qualitative analysis.
Through close reading of these posts and comments, we were able to build up a far more nuanced picture of the connections between relationships and financial hardship – which spanned everything from vital support, to abuse and extortion. When we brought the initial findings back to JRF and to the GPAG, these insights strongly resonated with their own expertise and lived experiences.
4. How could others approach this work?
Hopefully this outline can serve as an inspiration for those interested in developing their own mixed methods analysis! For more technical details, please see our full methodology in the report, or get in touch with [email protected], [email protected] or [email protected].