The Next Dangerous Fad in Machine Learning
By Mona Sloane, MIT Technology Review
August 26, 2020 -- The AI community is
finally waking up to the fact that machine learning can cause disproportionate
harm to already oppressed and disadvantaged groups. We have activists and
organizers to thank for that. Now, machine-learning researchers and scholars
are looking for ways to make AI more fair, accountable, and transparent—but
also, recently, more participatory.
One of the most exciting and
well-attended events at the International Conference on Machine Learning in
July was called “Participatory Approaches to Machine Learning.” This workshop
tapped into the community’s aspiration to build more democratic, cooperative,
and equitable algorithmic systems by incorporating participatory methods into
their design. Such methods bring those who interact with and are affected by an
algorithmic system into the design process—for example, asking nurses and
doctors to help develop a sepsis detection tool.
This is a much-needed intervention in
the field of machine learning, which can be excessively hierarchical and
homogenous. But it is no silver bullet: in fact, “participation-washing” could
become the field's next dangerous fad. That’s what I, along with my coauthors
Emanuel Moss, Olaitan Awomolo, and Laura Forlano, argue in our recent paper “Participation
is not a design fix for machine learning.”
Ignoring patterns of systemic oppression
and privilege leads to unaccountable machine-learning systems that are deeply
opaque and unfair. These patterns have permeated the field for the last 30
years. Meanwhile, the world has watched the exponential growth of wealth
inequality and fossil-fuel-driven climate change. These problems are rooted in
a key dynamic of capitalism: extraction. Participation, too, is often based on
the same extractive logic, especially when it comes to machine learning.
Participation isn’t free
Let’s start with this observation:
participation is already a big part of machine learning, but in problematic
ways. One way is participation as work.
Whether or not their work is
acknowledged, many participants play an important role in producing data that’s
used to train and evaluate machine-learning models. Photos that someone took
and posted are scraped from the web, and low-wage workers on platforms such as
Amazon Mechanical Turk annotate those photos to make them into training data.
Ordinary website users do this annotation too, when they complete a reCAPTCHA.
And there are many examples of what’s known as ghost work—anthropologist
Mary Gray’s term for all the behind-the-scenes labor that goes into making
seemingly automated systems function. Much of this participation isn’t properly
compensated, and in many cases it’s hardly even recognized.
Participation as consultation,
meanwhile, is a trend seen in fields like urban design, and increasingly in
machine learning too. But the effectiveness of this approach is limited. It’s
generally short lived, with no plan to establish meaningful long-term
partnerships. Intellectual-property concerns make it hard to truly examine
these tools. As a result, this form of participation is too often merely
performative.
More promising is the idea of
participation as justice. Here, all members of the design process work
together in tightly coupled relationships with frequent communication.
Participation as justice is a long-term commitment that focuses on designing
products guided by people from diverse backgrounds and communities, including
the disability community, which has long played a leading role here.
This concept has social and political importance, but capitalist market
structures make it almost impossible to implement well.
Machine learning extends the tech
industry’s broader priorities, which center on scale and extraction. That means
participatory machine learning is, for now, an oxymoron. By default, most
machine-learning systems have the ability to surveil, oppress, and coerce
(including in the workplace). These systems also have ways to manufacture
consent—for example, by requiring users to opt in to surveillance systems in
order to use certain technologies, or by implementing default settings that
discourage them from exercising their right to privacy.
Given that, it’s no surprise that
machine learning fails to account for existing power dynamics and takes an
extractive approach to collaboration. If we’re not careful, participatory
machine learning could follow the path of AI ethics and become just another fad
that’s used to legitimize injustice.
A better way
How can we avoid these dangers? There is
no simple answer. But here are four suggestions:
Recognize participation as work. Many
people already use machine-learning systems as they go about their day. Much of
this labor maintains and improves these systems and is therefore valuable to
the systems’ owners. To acknowledge that, all users should be asked for consent
and provided with ways to opt out of any system. If they chose to participate,
they should be offered compensation. Doing this could mean clarifying when and
how data generated by a user’s behavior will be used for training purposes (for
example, via a banner in Google Maps or an opt-in notification). It would also
mean providing appropriate support for content moderators, fairly
compensating ghost workers, and developing monetary or nonmonetary reward
systems to compensate users for their data and labor.
Make participation context specific.
Rather than trying to use a one-size-fits-all approach, technologists must be
aware of the specific contexts in which they operate. For example, when
designing a system to predict youth and gang violence, technologists
should continuously reevaluate the ways in which they build on lived experience
and domain expertise, and collaborate with the people they design for. This is
particularly important as the context of a project changes over time.
Documenting even small shifts in process and context can form a knowledge base
for long-term, effective participation. For example, should only doctors be
consulted in the design of a machine-learning system for clinical care, or
should nurses and patients be included too? Making it clear why and how certain
communities were involved makes such decisions and relationships transparent,
accountable, and actionable.
Learn from past mistakes. More
harm can be done by replicating the ways of thinking that originally produced
harmful technology. We as researchers need to enhance our capacity for lateral
thinking across applications and professions. To facilitate that, the
machine-learning and design community could develop a searchable database to
highlight failures of design participation (such as Sidewalk Labs’
waterfront project in Toronto). These failures could be cross-referenced with
socio-structural concepts (such as issues pertaining to racial inequality).
This database should cover design projects in all sectors and domains, not just
those in machine learning, and explicitly acknowledge absences and outliers. These
edge cases are often the ones we can learn the most from.
It’s exciting to see the
machine-learning community embrace questions of justice and equity. But the
answers shouldn’t bank on participation alone. The desire for a silver bullet
has plagued the tech community for too long. It’s time to embrace the
complexity that comes with challenging the extractive capitalist logic of
machine learning.
Mona Sloane is a sociologist based
at New York University. She works on design inequality in the context of AI
design and policy.
No comments:
Post a Comment