Taking human experience, which is beautifully ambiguous and nuanced and nondeterministic, and trying to fit it into a database shape, is inherently extractive. Nilay points out that it flattens people, which is totally true, but it also transfers ownership of that experience from their subjective truth into a centralized database that someone else controls, sets the standards for, and profits from.
This is balance nonprofits need to struggle with as they understand how and where to use AI. How do we keep the texture of experience? How do we keep the agency of it in our community?