AI is one of the most powerful inventions of our time right up there with the calculator, personal computer, and smartphone. But it’s still just a tool. A brilliant one, yes, but not perfect. To get the most out of it, you need to treat it in two distinct ways: as a teacher and as a collaborator.

1. Treat It as a Teacher

AI is the first tool that can teach you how to use it better. If you're unsure how to ask something, just ask the model itself:

“What’s the best way to phrase this question?”

This kind of self-improvement loop is unique to AI. Let it coach you on better prompting, formatting, or even how to approach a problem. Think of it as an interactive tutor that gets sharper with better questions.

2. Treat It as a Collaborator

AI works best when it has context, direction, and feedback, just like any good teammate. Don’t expect great results from vague or underspecified prompts. Instead, share your goals, experience, and constraints.

“I’ve spent X years in marketing and I’m focused on scaling product Y. Suggest five experiments to accelerate growth.”

Everyone has access to the same AI, but the quality of your results depends on how you interact with it. The better your input, the better the output.

Subscribe now

A Little 🤏🏼 Bit of Theory: How Does AI Really Work?

ChatGPT is powered by a large language model (LLM), built on a technology called the transformer architecture. At its core, the model works by predicting the next most likely word or rather, token in a sequence based on patterns it learned from vast amounts of text.

It doesn’t "think" in words. Instead, it uses tokens - chunks of words mapped to strings of 1s and 0s. For example, the word “You” is a single token, while “hippopotamus” breaks into multiple tokens.

👉 Here is a helpful app to see how human input gets translated into what the model actually sees:

https://tiktokenizer.vercel.app/

The model also processes more than just your words. Every ChatGPT conversation includes a hidden system prompt instructions set by developers that define how the model should behave. These instructions are separated from your input using special markers like <|im_start|>system<|im_sep|>.

Once it receives the full prompt (system + user), the model simply predicts what tokens are most likely to come next based on everything it has seen in training. That also means it can only generate responses based on information it already knows. For example, if a new country was founded yesterday, the model won’t be aware of it unless you include that info in your prompt.

💡 You can teach the model new things temporarily by including details in your prompt. Depending on the version, you can paste in hundreds or even thousands of pages of text. So while training data matters, your prompt is just as important.

As data scientists like to say: garbage in, garbage out.

And remember, even when it sounds smart, the model is just guessing the next likely words. It can still make mistakes (called hallucinations), so always double-check the output.

Prompts 101

To get the best results from AI, your input needs to be clear, well-structured, and intentional. Here’s one of the most important principles to keep in mind:

1. Placement Matters: Structure Your Prompt Strategically

The model pays the most attention to the beginning and the end of your prompt. Information placed in the middle is more likely to be overlooked especially in longer prompts.

To make sure your key points are noticed, follow this order:

• Start – Most important context or instructions

• Middle – Supporting or lower-priority details

• End – Key actions, goals, or final clarifications

This structure helps ensure the model focuses on what matters most and gives you more accurate, relevant responses.

2. Put Instructions First, and Use Clear Separators

Start your prompt with clear instructions, and separate them from the context using triple quotes (""") or hashtags (###). This helps the model understand where the instructions end and the input begins.

Less effective❌:

Summarize the text below as a bullet point list of the most important points. {text input here}

Better ✅:

Summarize the text below as a bullet point list of the most important points. Text: """{text input here}"""

3. Be Specific, Descriptive, and Detailed

The more specific you are about your desired outcome: context, format, tone, length, or style, the better the results. Vague prompts = vague answers.

Less effective ❌:

Write a poem about OpenAI.

Better ✅:

Write a short inspiring poem about OpenAI, focusing on the recent DALL-E product launch (DALL-E is a text to image ML model) in the style of a {famous poet}

4. Don’t Just Tell – Show

When asking for structured output, it’s far more effective to show the format you expect. Clear examples help the model follow instructions more accurately and make it easier to parse the results programmatically.

By clearly defining the format, you're not just guiding the model you’re also making the output more predictable and usable.

Less effective ❌:

Extract the entities mentioned in the text below. Extract the following 4 entity types: company names, people names, specific topics and themes.

Text: {text}

Better ✅:

Extract the important entities mentioned in the text below. First extract all company names, then extract all people names, then extract specific topics which fit the content and finally extract general overarching themes

Desired format:

Company names: <comma_separated_list_of_company_names>

People names: -||-

Specific topics: -||-

General themes: -||-

Text: {text}

5. Guide the Model with Examples: Zero-Shot vs. Few-Shot

The way you frame a task can make a big difference. You can either ask directly without examples (zero-shot) or provide a few examples first (few-shot) to show the model what you expect.

✅ Zero-shot Prompt.

Ask the model to perform a task without giving it any examples. This is quick but may be less reliable for complex tasks.

Extract keywords from the below text.

Text: {text}

Keywords:

✅ Few-shot Prompt.

Provide a couple of examples first. This helps the model learn the pattern and improves accuracy, especially for nuanced tasks.

Extract keywords from the corresponding texts below.

Text 1: Stripe provides APIs that web developers can use to integrate payment processing into their websites and mobile applications.

Keywords 1: Stripe, payment processing, APIs, web developers, websites, mobile applications

##

Text 2: OpenAI has trained cutting-edge language models that are very good at understanding and generating text. Our API provides access to these models and can be used to solve virtually any task that involves processing language.

Keywords 2: OpenAI, language models, text processing, API.

##

Text 3: {text}

Keywords 3:

Few-shot prompting works especially well when your task requires consistent formatting, interpretation, or categorization.

6. Be Clear - Cut the Fluff

Avoid vague or wishy-washy instructions. If you want precise results, give precise input.

Less effective ❌

The description for this product should be fairly short, a few sentences only, and not too much more.

Better ✅:

Use a 3 to 5 sentence paragraph to describe this product.

7. Don’t Say What Not to Do - Say What to Do Instead

Negative instructions alone aren’t enough. The model performs better when given a positive, constructive alternative.

Less effective ❌:

The following is a conversation between an Agent and a Customer. DO NOT ASK USERNAME OR PASSWORD. DO NOT REPEAT.

Customer: I can’t log in to my account.

Agent:

Better ✅:

The following is a conversation between an Agent and a Customer. The agent will attempt to diagnose the problem and suggest a solution, whilst refraining from asking any questions related to PII. Instead of asking for PII, such as username or password, refer the user to the help article www.samplewebsite.com/help/faq

Customer: I can’t log in to my account.

Agent:

8. For Code Generation: Use “Leading Words” to Set the Pattern

When prompting for code, start with keywords or syntax that signal the expected format. This primes the model for better results.

Less effective ❌:

# Write a simple python function that

# 1. Ask me for a number in mile

# 2. It converts miles to kilometers

Better ✅:

# Write a simple python function that

# 1. Ask me for a number in mile

# 2. It converts miles to kilometers

import

Leading with keywords like import, SELECT, or function nudges the model toward a specific output structure.

💡 For more advanced prompt engineering tips, check out promptingguide.ai/techniques.

ChatGPT Overview

Now that you’ve got the right mindset and a bit of theory under your belt, let’s dive into how to get the most out of ChatGPT in practice.

Let’s walk through the interface and key features to help you unlock its full potential.

① Message Composer & Action Bar

• Text field – Type your prompt here. Use ⌘ / Ctrl + Enter to submit.

• “+” Quick Action Button – Opens options to:

  • Upload files (PDFs, DOCX, images, CSVs, code, etc.). Up to 20 files and 100MB per message.

  • Take a photo / record a video (mobile only).

  Use this to expand context—for example, share internal documents, app design screenshots, or error messages. More context = better responses.

• Search – Turns your message into a live web query, enabling the model to access up-to-date information and cite sources. It also return results with  citation making responses easier to to verify.

• Deep research – A premium web‑research mode that hits a broader set of sources, bundles results, and lets the model reason over them before drafting an answer (think “research assistant” vs. “quick lookup”). It takes longer to complete as it reviews many sources, reflects, and then drafts a response. Despite the thorough process, it can still hallucinate or include misleading info—always review the output, and use citations to verify.

• Create image – Converts your prompt into an image. The more specific your request, the better the result. You can also upload a reference to influence style or layout. Note: one of the main weakness is correct text rendering in the images. When asking for text in the image have your expectation lower as text rendering in images is still limited.

• Voice Mode (sound waves icon) – Transcribe thoughts and make draft documents or emails. Great for a first draft when working on a lengthy document. You can even instruct the model to ask you follow up questions and keep guiding the conversation to have a discussion. On mobile, this mode can even use your camera to "see" and help with visual tasks (like, How to restart this router to get back my internet?).

② Left Sidebar: Navigation & Projects

• Chat List – Your recent chats. Pinned conversations stay at the top.

• Projects – Lightweight folders for organizing related chats and files. Each project can have its own:

  • Chat history

  • Custom instructions

  • Uploaded files (persistent across sessions)

  Great for ongoing research or client work, making it much easier to pick up the detailed conversation and requiring analysis where it was left.

• Explore GPTs – Discover and use custom GPTs made by others—or build your own. For example:

  • An internal HR assistant trained on your company policies, being able to quickly answer questions about PTO or learning budget without searching for specific document.

  • A brand voice assistant that helps everyone write on-brand.

• Sora – Access OpenAI’s video generation model (in supported versions).

• Search (magnifier icon) – Search across all your chats, files, and GPT responses.

③ Model Picker

Choose the right model for the task:

• GPT-4o – Fast and capable. Great for most use cases.

• GPT-4-turbo (o3) – More thoughtful and accurate, ideal for complex reasoning.

• GPT-4-mini – A solid fallback when usage limits are reached.

• GPT-4.5 (hidden under “More models”) – Known for high emotional intelligence. Some users find it overly expressive; OpenAI is actively tuning this.

🧠 Pro tip: Start with the fastest model. If the result isn’t good enough, switch to a more powerful one.

④ Workspace & Account Menu

• Tasks – A lightweight scheduler. For example: deliver a daily AI news summary at 8 AM.

• Customize ChatGPT – Teach the model about you:

  • Define your writing style

  • Set preferences (e.g., “No bullet points in summaries”)

  • Save time by avoiding repetitive instructions

  You can also update memory mid-prompt by saying, “Remember this…” and ChatGPT will confirm the memory update. Review and edit what the model remembers in the memory settings. Over time, this builds a more personalized and efficient assistant.

✅ You’re Ready to Start

There’s a learning curve but it’s a fun one. ChatGPT can handle repetitive tasks, help you strategize, or be your creative brainstorming partner. As you explore, you'll discover just how much of your workflow AI can support or even transform.

Subscribe now

Here’s the truth we need to say more often: LLMs are super capable - but also wildly unreliable. The difference between a flashy demo and a production-ready AI system is always reliability.

Let’s talk about how to build for it.

The AI Capability Rush — and the Reliability Cliff

With 68% of executives planning to invest up to $250M in AI this year, teams are under pressure to ship. The result? Demos that impress, systems that break. Most teams don’t lack ambition - they lack the operational muscle and experience to build systems that hold up under real-world conditions.

Let’s put numbers to that. Suppose your AI system has 10 steps: authenticate → fetch data → analyze → summarize → generate → and so on.

Each step has a 90% success rate. Sounds fine, right?

Until it doesn’t.

Your end-to-end reliability drops to just 35%.

Why? Because errors compound. One unreliable component can poison the entire system.

Browser-based agents like Manus or GPT Operator look slick in demos - until they have to scrape multiple pages or maintain task state. That’s why reliable systems need checkpoints: reset context, log every action, and validate outputs at every step.

Coding agents tend to perform better - not because they’re smarter, but because their outputs can be tested.

The pattern is clear: without validation at each step, even capable AI crumbles.

Authentication and Evaluation: The Foundation of Trust

Authentication isn’t a one-time checkbox - it’s a recurring process.

In agent systems where LLMs access user data, authentication failures create outsized risks. Each step that accesses data should have its own authentication verification - don't assume that because step 1 was authenticated, step 5 is still secure.

You don’t want to be the team that lets an LLM loose on customer data unchecked. Every action should re-auth, log its access, and be independently evaluated.

Trust the system not because it worked once - but because every step is verified, every time.

Binary evaluations make improvement possible

Subjective scoring doesn’t scale. Five people rating an output from 1–10 won’t give you actionable feedback. Replace it with binary criteria:

• Does it directly answer the user’s question? (Yes/No)

• Does it contain factual errors or hallucinations? (Yes/No)

• Is it compliant with our safety guidelines? (Yes/No)

• Does it match our brand voice? (Yes/No)

“3.3 vs. 4.7” means nothing. “Pass vs. Fail” tells you what to fix.

Human-in-the-Loop: Your Feedback Engine

Start your evaluation process with humans, ideally domain experts. Get real comments, identify common failure patterns, and only then begin automating parts of the evaluation. Don’t rush to replace people too soon. You’ll miss nuance, edge cases, and critical early insights.

This human-in-the-loop cycle is your feedback engine:

1. Experts provide ground-truth reviews

2. Patterns guide automation

3. Humans handle edge cases

4. Systems improve in a loop

Benchmarks Don’t Matter. Your Evaluation Framework Does

Don’t get distracted by flashy benchmark scores like MMLU, MATH, GPQA, HumanEval, or SimpleQA - they don’t predict how your model will perform in your real-world use case.

Every AI system needs a custom evaluation framework, tuned to its specific workflows and failure points. That’s why 80% of AI projects fail, even with strong third-party metrics. And you don’t need complex tools to get started - a simple spreadsheet tracking outputs, failures, and reliability step by step is often more valuable than any enterprise-grade platform. As you learn more, upgrade your tools - but never confuse sophistication with impact. Clarity beats complexity. Don’t over-engineer.

How to Start Building for Reliability

1. Map your AI workflows step by step

2. Define binary pass/fail criteria for each step

3. Set up a basic dashboard for per-step reliability

4. Start with heavy human involvement

5. Create graceful fallback paths for failures

Reliability isn’t a feature. It’s the product.

And it’s what makes your AI usable, safe, and trustworthy in the real world.

In my next post, I’ll break down how I built a lightweight email labeling tool - and the simple steps I took to make the data reliable from day one.

Hit subscribe to get the walkthrough.

Subscribe now

1. The hype hangover – AI feels like the next crypto. Flashy, confusing, and ultimately not that useful (yet).

2. The fear of obsolescence – “If this works, won’t I lose my job?”

These are real, human concerns. But the history of technology tells a different story. Take ATMs, for example. When they were introduced, everyone predicted bank tellers would disappear. But the opposite happened: bank employment actually grew, because ATMs reduced the cost of opening new branches, and more people started using banking services.

What’s happening inside companies

I recently spoke with a Principal Engineer at a Fortune 500 company with 30,000+ employees, most of them developers. Out of that massive pool? Only around 50–100 people are actively involved in AI-related initiatives or best-practice meetups.

That’s 0.3%.

Despite AI being on every slide deck and quarterly roadmap, the ground-level adoption is minimal.

Why?

“B2B clients require AI features, but refuse to enroll in beta test programs, so they turn to consultants and start develop something on their own”.

How to break the resistance (tips for businesses)

To shift perception, organizations need to make it really clear that AI isn’t about replacing people - it’s about helping them work better. (Think: 4-day workweeks, not pink slips.)

A few things that actually **work:

1. Create an AI ambassador network – Appoint 1–2 people in each function (design, product, marketing, dev, ops) to test-drive AI in real workflows. Let them become champions for their teams.

2. Start small with ROI-backed use cases – Pick a pain point that costs the business $10K+/month and try to solve it with AI. Tangible savings make skeptics curious.

3. Talk like a human, not a futurist – Don’t overpromise. AI isn’t going to run your company. But it can summarize meeting notes, write QA tests, or help a marketer ship 10x more content.

Not everyone can afford to ignore it (tips for individuals)

We’ve seen this before: modern farms, advanced factories, and digital businesses now produce far more with fewer people. But those displaced didn’t vanish - they transitioned, reskilled, or created new paths.

AI will do the same. Routine-heavy roles like basic QA, reporting, or customer support will shrink - not because they lack value, but because automation makes them more efficient.

At the same time, AI is the most accessible upskilling tool we’ve ever had. It can teach, coach, and co-create.

The transition won’t be painless. But early adopters gain the edge. Companies care about outcomes - more code, more ideas, more value. And the people who can deliver that at speed aren’t being replaced.

Every major technological revolution - from electricity to the internet - created more jobs than it destroyed. The challenge is always re-skilling fast.

Coders are skeptical, then hooked

AI is especially disruptive in software development. Many engineers resist it at first - they’re too busy, they’ve heard about hallucinations, they don’t trust it.

But that skepticism fades quickly after trying it. One teammate built an Android app over a weekend using AI, with zero prior experience. Another said it felt more like reviewing code than writing it.

AI shifts coding from "writing lines" to "reviewing, improving, guiding." Andrej Karpathy describes it as more collaborative and conversational than mechanical.

It won’t replace developers anytime soon, but it will accelerate them:

• Faster prototyping and experimentation

• Smoother entry into new languages

• More leverage for senior engineers

It removes the language barrier between human and machine. But you still need to know what you're building.

And then… there’s Apple.

One of the most surprising stories I heard recently: developers at Apple aren’t allowed to use AI tools - at all. Security concerns. So instead of co-pilots and smart refactoring, they’re still trawling Stack Overflow (which, by the way, is in serious decline, source).

In a race where AI becomes the new literacy, not allowing your team to use it is like asking them to write code with their hands tied.

It’s not AI that replaces you…

There’s a temptation to see AI as either a massive disruptor or an overhyped trend. But in reality, it’s something in the middle: a transformational tool with plenty of rough edges that requires time, practice, and humility to master.

The faster you experiment, the faster you find the value. And the longer you ignore it, the harder the transition becomes.

Because no, you’re not being replaced by AI.

You’re being replaced by someone who knows how to use it better.

There’s an urgent push from boards and executives to “do something with AI.” Nearly 99% of CEOs say they’re investing in generative AI this year, and 68% plan to funnel up to $250M into it.

The pressure is high, the budgets are huge - and the risk of waste is even bigger.

Here’s what’s happening: executives, driven by AI FOMO, are demanding fast results. Internal teams often lack the experience or resist change. As a result, companies turn to consultants who build flashy MVPs and promise ambitious outcomes. But once it’s time to move from prototype to production, it starts falling apart.

Why AI projects fail (a lot)

• AI is probabilistic, not deterministic. It doesn’t work like traditional software. You can’t treat it like a calculator where 2+2 always equals 4.

• Internal engineers are thrown into AI projects without the necessary experience, tooling, or data infrastructure.

• Getting to 80% is relatively easy. Getting from 80% to 95%? That’s the hard (and expensive) part.

• Leaders don’t always realize that MVP ≠ production-ready. This mismatch creates chaos, wasted money, and broken trust in AI.

We’ve seen this story before - remember the data science hype a decade ago? Everyone jumped in, then realized their infrastructure wasn’t ready. The same mistake is playing out again with AI.

If mishandled, this gap can erode trust in both AI initiatives and consulting practices- potentially setting back progress across the broader AI movement. And within the company, it often leads to frustration, disillusionment, and wasted resources.

And when it fails, it really fails:

• A car dealership chatbot sold a Chevy for $1. (VentureBeat)

• A lawyer used AI in court and cited fake cases. (Forbes)

• Even Apple faced heat for overpromising and underdelivering on AI features. (The Verge)

So what can we do about it?

1. Take a long-term view. Move fast - but not blindly. AI will reshape your business. Treat it like a transformation, not a side project.

2. MVP ≠ ready for production. Fast wins are great for learning. Don’t confuse them with something that can scale.

3. Upskill your teams. AI systems require new skills. Invest in talent who understand data pipelines, feedback loops, and AI-specific engineering.

4. Respect the nature of AI. AI’s probabilistic nature is a feature, not a flaw. Work with it - not against it.

5. Fix your data. Dynamic, contextual, and clean data is what fuels successful AI. Without it, even the best models won’t help you.

6. Failure is part of the journey. Over 80% of AI projects fail - often due to poor data, scope mismatch, or unrealistic expectations. That doesn’t mean AI is broken, your approach might be.

And yes, it’s possible to make it work

In one of my recent newsletters, I shared a story from my own experience - where taking an internal, strategic approach helped reduce projected annual costs from over $700,000 to under $50,000. Here’s the full breakdown and how we did it.

Or take Novo Nordisk as another example: they now generate 90% of Clinical Study Reports with AI, cutting a 12-week process down to just 10 minutes - without layoffs. They simply reallocated talent and built smartly. (Case Study)

There are more examples – I’ll share in my upcoming reviews. If you’re navigating these waters too, subscribe for more practical updates and lessons.

Subscribe now

Companies are moving fast to adopt AI - but beneath the excitement, questions remain. Quick wins are easy - lasting success is not.

Here are my six key takeaways from the conference - I’ll dive deeper into each one with specific examples, trends, and solutions in my upcoming newsletters.

#1 The AI gold rush is real — but it also backfires

There’s a massive push from boards and executives to adopt AI — and they want it done yesterday. According to a recent survey, 68% of executives plan to invest up to $250 million in AI this year (Forbes). That’s not even counting the massive AI infrastructure projects where companies are committing hundreds of billions of dollars.

But here’s the problem: while leadership demands quick action, internal teams often lack the expertise or structure to deliver. So, companies lean heavily on consultants who promise quick wins and deliver promising MVPs. But when it’s time to move from prototype to production, things start to get more complicated.

➡️ The leap from an 80% working MVP to a reliable, production-grade solution is huge - and most teams aren’t ready for that challenge.

➡️ If mishandled, this gap could lead to wasted resources, reputational damage, and growing skepticism about AI’s true potential. We’re seeing it already.

#2 Employees are skeptical — and fearful

Despite the top-down push for AI, many employees remain resistant. A Fortune 500 company with over 30,000 developers told me that only 50–100 people were actively involved in AI projects or knowledge-sharing sessions.

Why the resistance?

• Some see AI as overhyped - another tech bubble waiting to pop.

• Others fear AI will replace their jobs and make their skills obsolete.

But history suggests that AI is more likely to create opportunities than reduce them. When ATMs were introduced, many feared they would eliminate bank teller jobs - instead, banking employment grew as services expanded. AI could have a similar effect, making coding more accessible and increasing the demand for creative, strategic work.

I can’t think of any tech revolution that has reduced the overall number of jobs. Each time, we end up needing more people - the real challenge is the period of re-skilling.

#3 Data curation and evaluation are everything

AI models are only as good as the data they’re trained on - and data curation remains one of the most overlooked aspects of AI development.

➡️ Small errors scale fast. A minor data quality issue can ripple through the system, causing AI-generated insights or actions to fail spectacularly.

➡️ AI systems need continuous feedback loops and structured evaluation at every stage to improve over time.

➡️ Building a scalable, reliable AI system means putting data quality at the center - not treating it as an afterthought.

#4 AI Engineers are actually Reliability Engineers

LLMs are incredibly capable - but highly unreliable. AI engineers' job isn’t just to build models - it’s to make them consistent and predictable in real-world use cases.

➡️ Building a capable AI model is easy. Making it reliable at scale is hard.

➡️ AI engineers need to design systems that anticipate failure - and build in mechanisms to catch and correct issues in real time.

➡️ The difference between a successful AI rollout and a costly failure? A focus on reliability, not just capability.

#5: AI is becoming an active user of your product

AI isn’t just a backend tool anymore — it’s becoming an active user of your product. Businesses need to rethink not just how humans interact with their products, but how AI agents do as well.

➡️ Make it readable. Websites should have structured markdown files that are easy for AI to process - similar to how search engines crawl pages.

➡️ Design for AI. APIs should be built with AI agents in mind, using emerging standards like Model Context Protocol (MCP).

➡️ Search is changing. Google is already testing AI-generated search responses - essentially turning search into a dynamic, AI-generated interface.

#6: UX remains the missing link

AI still lacks intuitive user interfaces. Most AI-driven products are designed for engineers - not end-users.

➡️ AI needs a new generation of UX and UI thinking - interfaces that make AI capabilities intuitive, accessible, and actionable for everyday users.

➡️ The next wave of AI innovation will come not from better models - but from better interfaces.

If AI-generated search results and adaptable AI-powered products become the norm, UX will become the critical differentiator.

Interested in learning more? Over the next few weeks, I’ll break down each of these insights with practical solutions, best practices, and real-world examples. Hit subscribe and stay tuned.

Subscribe now

Here I share my recent AI deployment – a sentiment analysis and text rephrasing system designed to elevate user experience within a company previously reliant on rudimentary, rule-based sentiment tools.

I replaced a basic rule-based sentiment tool with a more nuanced AI solution, dropping annual costs from a projected $700,000+ to under $50,000. The shift preserved performance, strengthened data security, and created a user experience that feels both personalized and scalable.

TL;DR Framework: Scaling AI Solution in a Cost-Effective Way

• API-Driven Prototyping: Utilize third-party APIs for rapid concept validation and initial user feedback.

• Scalability Assessment & Cost Analysis: Analyze long-term API costs at scale. Recognize the inherent limitations and potential financial unsustainability of API-dependent solutions.

• In-House Expertise & Open-Source Adoption: Invest in building internal AI talent and embrace open-source models. This shift is crucial for long-term cost control, customization, and strategic advantage.

• Data-Driven Fine-Tuning: Leverage data collected during prototyping to create custom datasets for fine-tuning open-source models.

• Phased Deployment: Implement a phased rollout (beta programs, parallel testing) with rigorous validation and quality assurance at each stage.

• Continuous User Feedback: Embed mechanisms for ongoing user feedback to drive improvements.

Keep reading for step-by-step details.

The Starting Point: Rudimentary Systems and Rising Expectations

We began with a basic, rule-based sentiment system—functional, but too limited for modern needs. Users wanted not just sentiment detection, but also AI-driven text rephrasing aligned with company guidelines. Our objective: build an advanced AI solution for nuanced sentiment analysis and intelligent rewriting, all while integrating with legacy systems.

Phase 1: API Prototyping – Quick Validation

The best place to start AI exploration is with available third-party APIs. Their appeal is immediate: rapid prototyping and validation of core concepts. This approach let us quickly show internal stakeholders the impact and promise of these AI features. Low starting costs of the “pay-as-you-go” model, costing around $3-15 per million tokens, initially seem like a huge benefit for this experimental phase. It allowed us to quickly answer crucial questions: Is advanced sentiment detection valuable? Can AI-powered rephrasing improve user communication?

The answer to both was a resounding yes.

The API Bottleneck: Scalability vs. Cost

Most quickly realize that moving from prototyping to planning for broader deployment brings up the limitations of APIs.

Initial costs were manageable, but projections for scaling became alarming. For this case, estimates revealed that API costs for sentiment and rephrasing could exceed $700,000 annually – a prohibitive figure. This is mainly due to high throughput requirements needing dedicated server capacity and cutting-edge LLMs being too expensive for such tasks. Plus, the lack of customization options presented significant challenges. User feedback emphasized the need for personalized models and higher request throughput, functionalities difficult and costly to achieve with standard APIs.

While APIs were excellent for initial exploration, they were unsustainable for long-term, scalable solutions. So, what’s next?

Phase 2: Open-Source Advantage

Recognizing these limitations forced us to search for alternatives - building in-house AI solutions leveraging open-source models. The objective was to create a high-performance and economically viable solution at scale.

We turned to Hugging Face – a go-to hub for selecting the right model. After a quick review, we found that BERT-based models (RoBERTa, DistilBERT, ModernBERT) excel in sentiment analysis. Rigorous benchmarking, including cost-optimized solutions like AWS Inferentia, showed they offer both high performance and strong affordability.

The main hurdle with open-source models is fine-tuning them for specific needs—a step that requires curated data. Luckily, our API prototyping phase gave us exactly that. This fine-tuning was transformative, creating specialized, context-aware AI that outperforms generic APIs while slashing costs.

Phase 3: Ensuring Quality and Performance

When going to production, it is important to phase it carefully.

Rolling out the new in-house solution to a representative subset of users is the right thing to do. This controlled environment allows the assessment of real-world performance and iteration based on live usage patterns. Testing for performance is also crucial. As there was an existing legacy system that already served all users, we mirrored live traffic to our new model in “shadow mode.” This parallel approach, invisible to users, provided authentic performance data under real-world conditions.

Quality assurance was paramount. An internal review team evaluated the new model’s accuracy, quantifying improvements over the old rule-based system. To enhance efficiency, I also incorporated an LLM as a judge to augment human review, while maintaining critical oversight for long-term accuracy and stability.

Positive validation at each stage paved the way for wider beta releases and the introduction of enhanced features like AI-powered rephrasing, proactively assisting users in aligning communications with company standards.

Continuous Improvement: User Feedback as a Compass

Throughout beta and beyond, user feedback was our compass. Direct feedback mechanisms within the user interface, asking about model accuracy and helpfulness, provide continuous streams of insights and data for model improvements. Engaged users who provide extra comments created opportunities for a deeper understanding of user needs and ways to improve the experience.

The Cost Transformation

The financial impact of this transition was dramatic. Projected annual API costs of $700,000+ were transformed into in-house operational costs under $50,000 annually. This wasn’t just cost-cutting; it was a strategic shift.

If your AI pilot never evolves beyond a costly experiment, you’re missing out on genuine ROI. It’s easier—and cheaper—to integrate AI into your business than you might think, provided you have the right plan.

Subscribe to my newsletter for more real-world, cost-saving AI strategies.

Subscribe now