I tested ChatGPT's deep research capabilities with the most misunderstood law on the internet.

In the realm of generative artificial intelligence, the legal sector has proven to be one of the most problematic. Tools like OpenAI's ChatGPT have led to sanctions against lawyers and have embarrassed experts by generating documents based on fictional cases and citations of non-existent research. In light of this situation, the task was posed to evaluate ChatGPT's new "deep research" feature by applying it to a controversial topic in law: the interpretation of Section 230 of the Communications Decency Act.

ChatGPT was asked to compile a list of federal and Supreme Court rulings related to this section over the past five years and to summarize how these cases have been interpreted. This matter, which refers to the well-known "26 words that created the internet," is a constantly evolving topic of great interest. ChatGPT's response was positive in selecting and accurately summarizing a set of recent rulings; however, it overlooked some relevant points that a human expert might have considered and omitted an entire year of significant legal decisions that complicated the interpretation of the law.

The deep research feature aims to produce complex and sophisticated reports on specific topics, although accessing it in full requires a $200 monthly subscription. Unlike the basic version of ChatGPT, which has a limit on its training data, this tool seeks current information on the web, which should help present a more up-to-date analysis.

During the research process, ChatGPT asked follow-up questions to clarify the focus by requesting clarifications on specific areas of the law, but the final response did not address the broader context surrounding these decisions, which a human expert would likely have considered. Additionally, while the report covered from 2019 to 2024, the last decision mentioned was from 2023, giving the impression that there had been no new developments in the last year, despite significant developments occurring.

This lack of updates is critical, as in 2024 there was a ruling from the Third Circuit that limited the protections of the law to TikTok, which could have considerable implications for its future application. After reviewing the report, Eric Goldman, a legal expert, stated that despite some minor inaccuracies, the content was mostly correct, although he felt that ChatGPT did not capture the importance of certain cases well.

While OpenAI's deep research technology has shown significant progress, it has also revealed limitations in how it approaches the subject matter and in its ability to provide a complete narrative. The lack of context and details is a clear reminder that, although ChatGPT can offer informative summaries, it still relies heavily on human review to gather a more comprehensive understanding of the current legal landscape. The assumption that an artificial intelligence tool could replace expert analysis in law, especially in complex and rapidly changing areas, remains a challenge.