From Big Data to Big Theory: Lessons Learned from Archival Internet Research

Big data provides an opportunity to explore critical social science research questions in a new light, and to develop new records of historical events. In large part, these opportunities are due to the availability of new types of data such as digital traces, including online organizational records. For example, archived websites hosted at the Internet Archive provide insight into social and organizational phenomena on a previously unrealized scale. The HistoryTracker project aims to open up the Internet Archive for researchers, providing access to millions of archived web pages in a research-ready format. In addition, the theoretical grounding of this study underscores the connections that exist between Big Data research and ongoing work to develop new explanatory mechanisms driven by theory – in other words, Big Data must be complemented by Big Theory.

Thus, leveraging records accessed through the History Tracker project, this exemplar project studies the evolution of organizational phenomena by pairing archival web data with a theoretical framework grounded in evolutionary theory. In particular, this work focuses on examples of the evolution and growth of news media, and of social movements. Archival Internet data is utilized to illustrate macro-shifts in both organizational movements, examining the large-scale patterns of organizational adaptation through in response to new technological regimes.

The findings from this research are based on archival web data that tracks organizations emerging during the growth of the Occupy Wall Street movement, and the interplay between traditional news organizations and digital online-only providers. Growth and survival patterns were assessed based on hyperlinking patterns between key organizational populations. The findings present a nuanced perspective on the factors that impact the survival of new organizational forms. In aggregate, research from the HistoryTracker project demonstrates the power of large-scale data analysis when paired with theoretically driven research questions.