Prelude

Plastic influencers

Plastic influencer. AI Fanboy. Cardboard expert.

All terms entering the modern lexicon to describe the wave of ‘hype’ surrounding AI. I’ve long been a skeptic of some of the more outlandish and grandiose claims in the GenAI scene.

• 1/ Programmers will disappear
• 2/ AGI will arrive in 2024
• 3/ All jobs will be automated
• 4/ Robots will become conscious (Skynet)

All this baseless hyperbole without even delving into the more extremist views (there is a Reddit forum called singularity that has 3.4 million members).

I’m particularly bemused by the projection of emotion and fantasy onto computer algorithms capable of doing cool stuff. I believe that many brilliant people who subscribe to this Skynet perception of AI consciousness are at risk of losing their sanity. I’m not one of them.

My recent blogs have been in contradiction to the mainstream and somewhat fantastical AI world view

This preface is to make a point.

Believe me when I say that I don’t say this lightly.

In terms of building practical applications with GenAI, what Google has just done with Gemini 2.0 flash has changed absolutely everything. Everything.

And no one saw it coming.

A Christmas Nativity

How Open AI’s theatre became a pantomine

One of my parents favourite stories is how when I was 5 years old, I was given a part in the local nativity play. Cast as a tree, my role was to silently adorn the set while the older and more capable children performed their interpretation of the birth of Jesus Christ.

I wasn’t particularly happy with this minor role.

Over the next 10-15 minutes, I followed the cast about stage, stealing their lines and thundering out my own entirely different interpretation of the play.

Interjecting at perfect moments, performing at others. It was a masterclass of disruption, and every giggle and teary eye from the watching crowd goaded me into more. It was ruthless destruction.

The performance descended into farce, the audience crying with laughter; the actors bemused and confused.

The laughter encouraged me, it became a crescendo.

The play was converted into pantomime, the job complete. To this day it remains a tale told at dinner parties to new and younger family members.

Of course, the play unfolding in Christmas 2024 is Open AI’s 12 days of Christmas and how Google has not just stolen their thunder, but commandeered the narrative, stolen the limelight and turned a Christmas celebration from OpenAI into a winter nightmare.

I, (like most rational people), tuned into the 12 days of Christmas by OpenAI with a healthy degree of skepticism, and watched as they showed demos of phone calls and astronomically expensive & slow API calls to a marginally improved LLM model, and felt reassured that my cynical world view was validated.

Then something happened.

It happened with perfect timing; theatre at it’s best.

Like an earthquake the repercussions are coming and they will be felt by everyone and seen in every AI product in the near future.

I thought Google had dropped the ball on AI, we all did. They were just irrelevant in all practical usages. Quality was poor, functionality was limited.

It turns out that they didn’t drop the ball and they weren’t asleep on the job. They were simply leaving the competition (now children by comparison) to wrestle with Beta releases, barely functioning APIs and scale issues while quietly building the tooling that is necessary to effectively use GenAI in production.

They timed their entrance to maximum effect.

Until a week ago I didn’t even have a live Google API Key.

This week, I’m in the process of migrating every single one of my services.

This may seem rash, but let me explain.

Scientists vs Builders

The difference between theory and practice

There are two different factions within the world of AI right now; scientists and builders.

The pioneers and scientists are seeking AGI and novel use cases; this includes important work such as new approaches to cancer treatments or looking for academic breakthroughs in Quantum physics. This can be theoretical or even in some cases some green shoots of practical use cases, especially in the domain of robotics for example.

These folk are interested in pursuing AGI and adapting GenAI to a more hybrid form of intelligence that will exponentially increase utility over current LLMs. This may take years, it may take generations (probably!).

I’m firmly and unashamedly in the second faction; we are builders.

GenAI is already capable of incredible stuff. Things that a year or two ago would have been impossible. I want to build stuff that works, right now.

The job at hand is working with available LLMs and APIs and seeing what use cases we can implement.

A builder needs tools and my stack was derived from countless hours spent testing the utility of all the available APIs and models.

• 1/ Claude 3.5 Sonnet for Coding (Code)
• 2/ OpenAI APIs for structured data w/ reasoning (Orchestration)
• 3/ Groq / Fireworks AI APIs for cheap and instant inference (Fast inference)
• 4/ Llama for local/on device (Edge computing)

I thought this stack would be solid for the next 3-5 years.

To be honest, I wasn’t really interested in any GenAI model that wasn’t listed above, I wasn’t even paying attention to the Gemini Flash v2.0.

I’m paying attention now.

How Agents work

2025, the year of the Agent.

We all know that 2025 is the year of the Agents, social media won’t stop telling us.

I hate hype trains but the underlying truth is that AI systems are now basically capable of ‘semi-reliably’ taking actions on our behalf. Thus, it is fair to say that there will be loads of popular software released in 2025 that will use this paradigm.

A typical agentic flow goes something like this.

We receive an instruction (Book a flight, call my mum, make my breakfast) which is interpreted by a Prompt. A prompt is usually executed via API, hence your OpenAI or Groq or Fireworks AI API). That prompt calls a tool (Skyscanner, Web search) which gets the result and calls some code setup by the developer and does “stuff”.

The result of this “stuff” is then returned to another Prompt and the cycle continues (nJumps) until we have performed the action. Hurrah.

It doesn’t look like the cleanest architecture does it?

If any of these API calls fails or returns an unexpected result, the whole chain is broken. Dozens of Python Frameworks have emerged to abstract this problem, but they can’t solve it. Tooling is improving, we can now see errors in execution, validate structured data and build chains with something approaching reliability, hence the hype for Agent 2025.

But the above architecture remains convoluted, complex and unreliable. Despite this, it is also the only way we had to unlock the potential of GenAI in Agentic flows.

In Dec 2024 Google has just made the above agentic model obsolete before it has even become ubiquitous.

The primary reasons are as follows:

• 1/ Native search
• 2/ Integrated orchestration
• 3/ Multi-modal (which works!)

Google vs OpenAI & Perplexity

Native tooling: Search that works

Have a read of the Gemini API docs, and bear in mind that this isn’t a proposal or a fantasy, but an API that works and can provide results in milliseconds.

Google’s integrated search is reliable and also works quickly. Rivals such as Perplexity have a text based AI search engine, it has its place in the wider landscape but bear in mind that this the value proposition of an AI Unicorn has now been integrated as a ‘feature’ of Gemini Flash v2.0.

Perplexity AI’s purpose and reason for existence has been assumed within an actual AI model that is capable of the same quality and speed of result with massive utility in other areas as well.

The fact that Google owns a proprietary Search engine is critical here. They have a genuinely “Native Tool” in every sense, bundled into the same API serving the inference model that can search the internet available by just adding some text to the API call.

Ah, but OpenAI can do that too I hear you say?

OpenAI can’t compete. Their search is not native (or not mature) and that is important. It really shows. They have a “Realtime API”, but it doesn’t work that well and is noticeably slower and buggier than Google’s Gemini Flash v2.0 implementation. In real time more than any other domain, latency is everything. The results are not even close.

OpenAI interaction Example

,

Google vs Python Frameworks

Simple Agentic flows: RIP Python abstractions.

But it doesn’t stop there. Google has changed the game in terms of Agentic flows. Search the internet for “AI Tools” and you will find mountains of frameworks, code repos and projects that are basically doing the same thing.

• 1/ Search Internet; Check
• 2/ Scape website; Check
• 3/ Convert to markdown; Check
• 4/ Run code; Check

All these tools are automating search, retrieval and code execution. Have a look at the Langchain Tools for example.

The thing is, Google has just integrated this into their API, a single endpoint to handle all of the above. It is now essentially a solved problem. We no longer need complex agentic flows for many many use cases.

The below diagram from OpenAI shows how function calling works for Agents.

Until now, we have been relying on managing the execution environment outside of the API calls.

Google has just built most of that functionality into a core API that can be used by developers.

For example, if I want to use Llama 3.3 to search the internet, I can do tool calling as follows.

This same flow with Gemini Flash v2.0:

Back to the previous point, Speed, Quality, Cost…

Google just chose all 3.

Nearly all agents are variations of search, retrieval (convert to markdown and inject into prompt) and arbitrary code execution with a sprinkling of private data. Except for the data (almost definitely coming soon…), these are now core concerns, which has made a lot of Agentic systems obsolete before they have been launched.

It won’t be long before we also have native plugins to your Google data sources (a logical next step), at which point except for a rare few scaled and highly complex AI systems, basically all the current frameworks and processes are just convoluted implementations of what can be achieved better, faster and cheaper in a single API call.

The relevance of this from an architectural point of view, is that instead of building chained and complex flows, I can refine a single simple model. Everything just became a lot simpler

Even if we can’t do everything we need right now, the line in the sand has been drawn and most common “tools” must become core concerns, integrated into APIs by providers. We don’t need to DIY our own Agents anymore, we have reliable, scaled and fast APIs to work with.

Bye bye Python frameworks. (don’t stay in touch).

Multi-Modal that works

Magical human to machine UX

Like me, you are probably a bit burned by all the multi-modal ‘demo’ examples of Audio/Video usage. I remember being so excited to try audio-streaming (I’ve been developing on WebRTC for years and in a past life founded an eCommerce video streaming tool).

The potential is obvious, but the whole thing just doesn’t feel right. For an example go to the OpenAI playground and try out their realtime API. It shows potential, but is miles away from being an enjoyable user experience. Most users just want an experience that “works”. Those milliseconds and natural intonations are not details, they are the very essence of the product.

Gemini Flash v2.0 is the first model that gave me the “wow” moment that I had when I first started using Claude for coding. It is the same feeling as the first time you sceptically asked ChatGPT a question and the “machine” gave you a human response.

The latency, the pauses, the voice intonation. Google has NAILED it. It is still obviously an AI system, but that was never the problem. The problem was always the pauses, the interruptions, the way that the model interacted with humans.

I don’t mind talking to a machine, assuming the machine is knowledgeable, able to interact and capable of doing the things that I need it to do. This is 100% the first time I’ve actually seen a model capable of providing this experience, and the ramifications are tremendous.

If you were excited by audio or video interactions and a bit sceptical of the models. Go give Gemini Flash v2.0 a try. Google has obviously invested time, effort and resources into solving issues about latency and cost. No other AI model that I have tried even comes close.

Conclusion

There was a dream that was the UX of Generative AI.

I’m as excited as the first time that I asked ChatGPT to write a linkedin post all those years ago. At this stage of my life and involvement with GenAI, that isn’t particularly easy.

I didn’t expect this moment to come so soon.

We now have a reality with a cheap, fast and highly capable model that we can interact with in real time.

This is literally the first time in my life that I can speak to a computer, and feel like it understands me, can respond to me, and take actions on my behalf. It isn’t a complex agent, it is a single API call.

This is a technical achievement that will reverberate through the AI world, even if many haven’t yet realised.

Apart from the natural interface and interactions, the model is capable of natively searching the internet, executing code and giving me the response in the time it takes to form a sentence.

There was a dream that was the UX of Generative AI.

In December 2024 it became a reality.

And it’s cheap…

And it’s scalable…

Now if you will excuse me, I’m off to build stuff.

Follow me: