Intent and sensemaking
The result is systems that can produce text that is very compelling when we as humans make sense of it. But the systems do not have any understanding of what they are producing, any communicative intent, any model of the world, or any ability to be accountable for the truth of what they are saying. … When people seek information, we might think we have a question and we are looking for the answer, but more often than not, we benefit more from engaging in sense-making: refining our question, looking at possible answers, understanding the sources those answers come from and what perspectives they represent, etc. Consider the difference between the queries: “What is 70 degrees Fahrenheit in Celcius?” and “Given current COVID conditions and my own risk factors, what precautions should I be taking?”
From All-knowing machines are a fantasy by Emily M. Bender and Chriag Shah.