Google's Gemini AI Overpromises: Struggles with Long Context and Data Analysis Accuracy

Both Gemini 1.5 Pro and 1.5 Flash models struggle to answer questions about large datasets correctly. In document-based tests, the models gave the right answer only 40% to 50% of the time. Specifically, Gemini 1.5 Pro answered true/false statements correctly 46.7% of the time, while Flash answered correctly only 20% of the time. These models have difficulty verifying claims that require considering larger portions of the text or implicit information that is clear to human readers but not explicitly stated.

The newest versions of Gemini can take in upward of 2 million tokens as context, which is equivalent to roughly 1.4 million words, two hours of video, or 22 hours of audio. This is the largest context of any commercially available model. However, despite this large context window, the models have not demonstrated a high accuracy in understanding and reasoning over such large contexts.

Researchers believe Google is overpromising with Gemini's capabilities because the models have not performed well in tests that measure their ability to understand and reason over large datasets. Studies have shown that Gemini models struggle with tasks that require understanding larger portions of text or implicit information. Additionally, other models like OpenAI’s GPT-4 and Anthropic’s Claude 3.5 Sonnet also did not perform well, but Google has been the only provider to heavily advertise the context window as a key feature.

Researchers suggest better benchmarks and greater emphasis on third-party critique to improve the evaluation of generative AI models like Gemini. Current benchmarks, such as the 'needle in the haystack' test, only measure a model’s ability to retrieve specific information rather than answer complex questions about that information. A more robust and standardized benchmarking culture is needed to accurately assess the capabilities of these models.