X, formerly known as Twitter, used to be a crucial tool for researchers and technology developers in their effort to take the pulse of society. Its open Application Programming Interface (API) used to provide vast amounts of data for analysis on various topics and fields, including disinformation spread. During February, X announced that it will stop supporting free access to the X API [1] launching paid tiers subscription that ranges from $100 (for 50,000 post requests and 10,000 read requests per app per month) to an outrageously expensive $42.000 per month for enterprise accounts. As it can be easily seen, the cost for collecting data from X is forcing researchers and developers to stop their projects [2] or try to find alternative data sources.
Restricted API Access Is Trending
The same restrictions imposed by X are also considered by other social media platforms. Reddit announced that it will start limiting third-party access to its API through additional terms and conditions.[3] Adding to this list the well-known API restrictions and shutdowns rolled out by Facebook in 2018[4], research efforts, particularly in the realm of social media analysis and combating of disinformation, had significant issues such as limited ability to track the spread of information, monitor public sentiment, check the credibility of shared information and assess the authenticity of content. On top of these events, there are not that many options for collecting bulk data from social media platforms. Scraping data from a website without the use of an API may be considered as an option but it requires a lot of work and can be fraught with other risks, especially concerning its ethical part. Nowadays, it is very hard to discern if an open-source intelligent platform is scraping information in the public interest for any malicious purposes (e.g., phishing).
How FERMI Copes With Limited Access to APIs
To cope with the recent changes in access to API of social media platforms, the technical team of FERMI considered several strategies in an attempt to ensure the validity of its outcomes. First, manual data scraping of publicly available content from Facebook and X was examined which seems like a feasible option for small-scale research despite its time-consuming process, provided that all data collection and analysis processes are legally and ethically compliant. Then, the project partners searched for other suitable social media platforms with accessible API such as Instagram (certain features), YouTube et cetera which according to some cases might do the work. Moreover, there was a concrete effort to connect with open data initiatives by governments or organisations that might release datasets related to social media and disinformation. These datasets provided valuable insights for the proof-of-concept of the platform’s technical modules. Lastly, the consortium reached out to other European Union funded projects in the field of disinformation for any dataset containing information related to the three use cases of the project. A combination of the aforementioned actions in accordance with legal and ethical requirements gave FERMI the opportunity to continue its efforts towards offering a holistic platform for combating the violent ramifications of fake news and disinformation through cutting-edge technologies fuelled by social media data. Nevertheless, the landscape of data access and availability is constantly evolving and our research teams need to stay updated with any changes in policies and terms of use of different social media platforms, exploring innovative methods and partnerships to gather data responsibly and effectively.
--
[1] Jenae Barnes, “Twitter Ends Its Free API: Here’s Who Will Be Affected,” Forbes, February 6, 2023, https://www.forbes.com/sites/jenaebarnes/2023/02/03/twitter-ends-its-free-api-heres-who-will-be-affected/.
[2] Matt Novak, “Twitter Makes It Harder To Build Third Party Apps And Fun Bots In Latest Cash Grab,” Forbes, February 2, 2023, https://www.forbes.com/sites/mattnovak/2023/02/02/twitter-makes-it-harder-to-build-third-party-apps-and-fun-bots-in-latest-cash-grab/.
[3] “Creating a Healthy Ecosystem for Reddit Data and Reddit Data API Access,” Reddit, April 18, 2023, https://www.redditinc.com/blog/2023apiupdates.
[4] Sarah Perez, “Facebook rolls out more API restrictions and shutdowns,” TedCrunch+, July 2, 2018, https://techcrunch.com/2018/07/02/facebook-rolls-out-more-api-restrictions-and-shutdowns/.