Selected ongoing projects
Evaluating Generative AI Systems is a Social Science Measurement Challenge.
https://arxiv.org/pdf/2411.10939
NeurIPS EvalEval workshop & SFLLM workshop
Hanna Wallach, Meera Desai, Nicholas Pangakis, A Feder Cooper, Angelina Wang, Solon Barocas, Alexandra Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z Jacobs.
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice.
https://arxiv.org/pdf/2412.06966
A Feder Cooper, Christopher A Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
Legitimacy and the Algorithmic Turn in the Administrative State. Presented at the Privacy Law Scholars Conference (PLSC) 2024.
Amina Abdu, A. Z. Jacobs
Notre Dame-IBM Tech Ethics Lab grant. (2022-2023) “Expanding AI Audits To Include Instruments: Accountability, Measurements, and Data in Motion Capturing Technology.” Co-PI with Mona Sloane (New York University), Emanuel Moss (Intel Research).
Scientific argument with supervised learning. NeurIPS Workshop on AI for Science: Mind the Gaps. [working draft]
Jeff Lockhart, A. Z. Jacobs.
Publications
Algorithmic Transparency and Participation through the Handoff Lens: Lessons Learned from the U.S. Census Bureau’s Adoption Differential Privacy.
ACM Conference on Fairness, Accountability and Transparency (FAccT), 2024. [pdf]
Amina Abdu, Lauren Chambers, Deirdre K. Mulligan, A. Z. Jacobs
An archival perspective on pretraining data.
Patterns, 2024. [pdf]
Meera A. Desai, Irene V. Pasquetto, A. Z. Jacobs, Dallas Card
The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology.
ACM Conference on Human Factors in Computing Systems (CHI), 2024, to appear. [pdf]
Emma Harvey, Hauke Sandhaus, A. Z. Jacobs*, Emanuel Moss*, Mona Sloane*
︎︎︎ ACM Best Paper Honorable Mention
︎︎︎ coverage by IEEE Spectrum: “AI Is Being Built on Dated, Flawed Motion-Capture Data“
The Role of Relevance in Fair Ranking.
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2023. [pdf]
Aparna Balagopalan, A. Z. Jacobs, Asia J. Biega
︎︎︎ popular writeup for the Montreal AI Ethics Institute
An empirical analysis of racial categories in the algorithmic fairness literature.
ACM Conference on Fairness, Accountability and Transparency (FAccT), 2023. [pdf]
Amina Abdu, Irene V. Pasquetto, A. Z. Jacobs
︎︎︎ popular writeup for the Montreal AI Ethics Institute
Conceptualizing Algorithmic Stigmatization.
ACM Conference on Human Factors in Computing Systems (CHI), 2023. [pdf]
Nazanin Andalibi, Cassidy Pyle, Kristen Barta, Lu Xian, A. Z. Jacobs, Mark Ackerman.
A large-scale comparative study of informal social networks in firms.
Management Science. (2021). [publisher link, open access pdf]
A. Z. Jacobs, Duncan Watts.
Measurement and fairness.
ACM Conference on Fairness, Accountability and Transparency (FAccT). (2021). [arXiv, related tutorial, slides, tutorial video]
A. Z. Jacobs, Hanna Wallach.
Internet-human infrastructures: Lessons from Havana’s StreetNet.
WebConf Workshop on Innovative Ideas in Data Science, nominated for best paper. (2020). [preprint]
A. Z. Jacobs, Michaelanne Dye Thomas (joint work).
Translation tutorial: The meaning and measurement of bias: Lessons from natural language processing
ACM Conference on Fairness, Accountability and Transparency (FAccT). (2020). [ACM link, related tutorial, slides, tutorial video]
A. Z. Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach.
Assembly in populations of social networks.
CSCW Workshop on Multi-Site Research. Short paper. (2018). [arXiv]
A. Z. Jacobs.
Assembling thefacebook: Using heterogeneity to understand online social network assembly.
ACM Web Science Conference (WebSci). (2015). [arXiv, data]
A. Z. Jacobs, Samuel F. Way, Johan Ugander & Aaron Clauset.
Learning latent block structure in weighted networks. Journal of Complex Networks 3(2), 221-248. (2015). [arXiv]
Christopher Aicher, A. Z. Jacobs & Aaron Clauset.
A unified view of generative models for networks: models, methods, opportunities, and challenges. NIPS Workshop on Networks: From Graphs to Rich Data. (2014). [arXiv]
A. Z. Jacobs & Aaron Clauset.
Efficiently inferring community structure in bipartite networks. Physical Review E 90, 012805. (2014). [arXiv, code]
Daniel B. Larremore, Aaron Clauset & A. Z. Jacobs.
Complex life cycles in a pond food web: effects of life stage structure and parasites on network properties, trophic positions and the fit of a probabilistic niche model. Oecologia 174 (3) 953-965. (2014). [publisher link]
Daniel L. Preston, A. Z. Jacobs, Sarah A. Orlofske & Pieter T.J. Johnson.
Adapting the stochastic block model to edge-weighted networks. ICML Workshop on Structured Learning (SLG). (2013). [arXiv, code]
Christopher Aicher, A. Z. Jacobs & Aaron Clauset.
Detecting friendship within dynamic online interaction networks. AAAI Conference on Weblogs and Social Media (ICWSM). (2013). [arXiv]
Sears Merritt, A. Z. Jacobs, Winter Mason & Aaron Clauset.
See also:
The Hidden Governance of AI. The Regulatory Review (2022). [link]
A. Z. Jacobs, Deirdre Mulligan.
Report of the 1st Workshop on Generative AI and Law.
Cooper, A. Feder and Lee, Katherine and Grimmelmann, James and Grimmelmann, James and Ippolito, Daphne and Callison-Burch, Christopher and Choquette-Choo, Christopher A. and Mireshghallah, Niloofar and Brundage, Miles and Mimno, David and Choksi, Madiha Zahrah and Balkin, Jack M. and Carlini, Nicholas and De Sa, Christopher and Frankle, Jonathan and Ganguli, Deep and Gipson, Bryant and Guadamuz, Andres and Harris, Swee Leng and Jacobs, Abigail and Joh, Elizabeth E. and Kamath, Gautam and Lemley, Mark A. and Matthews, Cass and McLeavey, Christine and McSherry, Corynne and Nasr, Milad and Ohm, Paul and Roberts, Adam and Rubin, Tom and Samuelson, Pamela and Schubert, Ludwig and Vaccaro, Kristen and Villa, Luis and Wu, Felix T. and Zeide, Elana.
Yale Law & Economics Research Paper, (November 16, 2023). [pdf]
Measurement as governance in and for responsible AI. KDD Workshop on Responsible AI (2021) [working draft]
A. Z. Jacobs
Comparative, population-level analysis of social networks in organizations. Dissertation. 2017.
Untangling the roles of parasites in food webs with generative network models. Preprint. (2015). [arXiv]
A. Z. Jacobs, Jennifer A. Dunne, Cristopher Moore & Aaron Clauset.
Adapting to non-stationarity with growing expert ensembles. Preprint. (2011). [arXiv]
Cosma R. Shalizi, A. Z. Jacobs, Kristina L. Klinkner & Aaron Clauset.
Evaluating Generative AI Systems is a Social Science Measurement Challenge.
https://arxiv.org/pdf/2411.10939
NeurIPS EvalEval workshop & SFLLM workshop
Hanna Wallach, Meera Desai, Nicholas Pangakis, A Feder Cooper, Angelina Wang, Solon Barocas, Alexandra Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z Jacobs.
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice.
https://arxiv.org/pdf/2412.06966
A Feder Cooper, Christopher A Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
Legitimacy and the Algorithmic Turn in the Administrative State. Presented at the Privacy Law Scholars Conference (PLSC) 2024.
Amina Abdu, A. Z. Jacobs
Notre Dame-IBM Tech Ethics Lab grant. (2022-2023) “Expanding AI Audits To Include Instruments: Accountability, Measurements, and Data in Motion Capturing Technology.” Co-PI with Mona Sloane (New York University), Emanuel Moss (Intel Research).
Scientific argument with supervised learning. NeurIPS Workshop on AI for Science: Mind the Gaps. [working draft]
Jeff Lockhart, A. Z. Jacobs.
Publications
Algorithmic Transparency and Participation through the Handoff Lens: Lessons Learned from the U.S. Census Bureau’s Adoption Differential Privacy.
ACM Conference on Fairness, Accountability and Transparency (FAccT), 2024. [pdf]
Amina Abdu, Lauren Chambers, Deirdre K. Mulligan, A. Z. Jacobs
An archival perspective on pretraining data.
Patterns, 2024. [pdf]
Meera A. Desai, Irene V. Pasquetto, A. Z. Jacobs, Dallas Card
The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology.
ACM Conference on Human Factors in Computing Systems (CHI), 2024, to appear. [pdf]
Emma Harvey, Hauke Sandhaus, A. Z. Jacobs*, Emanuel Moss*, Mona Sloane*
︎︎︎ ACM Best Paper Honorable Mention
︎︎︎ coverage by IEEE Spectrum: “AI Is Being Built on Dated, Flawed Motion-Capture Data“
The Role of Relevance in Fair Ranking.
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2023. [pdf]
Aparna Balagopalan, A. Z. Jacobs, Asia J. Biega
︎︎︎ popular writeup for the Montreal AI Ethics Institute
An empirical analysis of racial categories in the algorithmic fairness literature.
ACM Conference on Fairness, Accountability and Transparency (FAccT), 2023. [pdf]
Amina Abdu, Irene V. Pasquetto, A. Z. Jacobs
︎︎︎ popular writeup for the Montreal AI Ethics Institute
Conceptualizing Algorithmic Stigmatization.
ACM Conference on Human Factors in Computing Systems (CHI), 2023. [pdf]
Nazanin Andalibi, Cassidy Pyle, Kristen Barta, Lu Xian, A. Z. Jacobs, Mark Ackerman.
A large-scale comparative study of informal social networks in firms.
Management Science. (2021). [publisher link, open access pdf]
A. Z. Jacobs, Duncan Watts.
Measurement and fairness.
ACM Conference on Fairness, Accountability and Transparency (FAccT). (2021). [arXiv, related tutorial, slides, tutorial video]
A. Z. Jacobs, Hanna Wallach.
Internet-human infrastructures: Lessons from Havana’s StreetNet.
WebConf Workshop on Innovative Ideas in Data Science, nominated for best paper. (2020). [preprint]
A. Z. Jacobs, Michaelanne Dye Thomas (joint work).
Translation tutorial: The meaning and measurement of bias: Lessons from natural language processing
ACM Conference on Fairness, Accountability and Transparency (FAccT). (2020). [ACM link, related tutorial, slides, tutorial video]
A. Z. Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach.
Assembly in populations of social networks.
CSCW Workshop on Multi-Site Research. Short paper. (2018). [arXiv]
A. Z. Jacobs.
Assembling thefacebook: Using heterogeneity to understand online social network assembly.
ACM Web Science Conference (WebSci). (2015). [arXiv, data]
A. Z. Jacobs, Samuel F. Way, Johan Ugander & Aaron Clauset.
Learning latent block structure in weighted networks. Journal of Complex Networks 3(2), 221-248. (2015). [arXiv]
Christopher Aicher, A. Z. Jacobs & Aaron Clauset.
A unified view of generative models for networks: models, methods, opportunities, and challenges. NIPS Workshop on Networks: From Graphs to Rich Data. (2014). [arXiv]
A. Z. Jacobs & Aaron Clauset.
Efficiently inferring community structure in bipartite networks. Physical Review E 90, 012805. (2014). [arXiv, code]
Daniel B. Larremore, Aaron Clauset & A. Z. Jacobs.
Complex life cycles in a pond food web: effects of life stage structure and parasites on network properties, trophic positions and the fit of a probabilistic niche model. Oecologia 174 (3) 953-965. (2014). [publisher link]
Daniel L. Preston, A. Z. Jacobs, Sarah A. Orlofske & Pieter T.J. Johnson.
Adapting the stochastic block model to edge-weighted networks. ICML Workshop on Structured Learning (SLG). (2013). [arXiv, code]
Christopher Aicher, A. Z. Jacobs & Aaron Clauset.
Detecting friendship within dynamic online interaction networks. AAAI Conference on Weblogs and Social Media (ICWSM). (2013). [arXiv]
Sears Merritt, A. Z. Jacobs, Winter Mason & Aaron Clauset.
See also:
The Hidden Governance of AI. The Regulatory Review (2022). [link]
A. Z. Jacobs, Deirdre Mulligan.
Report of the 1st Workshop on Generative AI and Law.
Cooper, A. Feder and Lee, Katherine and Grimmelmann, James and Grimmelmann, James and Ippolito, Daphne and Callison-Burch, Christopher and Choquette-Choo, Christopher A. and Mireshghallah, Niloofar and Brundage, Miles and Mimno, David and Choksi, Madiha Zahrah and Balkin, Jack M. and Carlini, Nicholas and De Sa, Christopher and Frankle, Jonathan and Ganguli, Deep and Gipson, Bryant and Guadamuz, Andres and Harris, Swee Leng and Jacobs, Abigail and Joh, Elizabeth E. and Kamath, Gautam and Lemley, Mark A. and Matthews, Cass and McLeavey, Christine and McSherry, Corynne and Nasr, Milad and Ohm, Paul and Roberts, Adam and Rubin, Tom and Samuelson, Pamela and Schubert, Ludwig and Vaccaro, Kristen and Villa, Luis and Wu, Felix T. and Zeide, Elana.
Yale Law & Economics Research Paper, (November 16, 2023). [pdf]
Measurement as governance in and for responsible AI. KDD Workshop on Responsible AI (2021) [working draft]
A. Z. Jacobs
Comparative, population-level analysis of social networks in organizations. Dissertation. 2017.
Untangling the roles of parasites in food webs with generative network models. Preprint. (2015). [arXiv]
A. Z. Jacobs, Jennifer A. Dunne, Cristopher Moore & Aaron Clauset.
Adapting to non-stationarity with growing expert ensembles. Preprint. (2011). [arXiv]
Cosma R. Shalizi, A. Z. Jacobs, Kristina L. Klinkner & Aaron Clauset.