Finding a good partner to build your next generative AI project
When you're looking to partner with a professional services firm to build a generative AI solution, it's crucial to ensure that they have the right mix of skills and approaches to meet your specific needs.
With the rapid advancements in AI technology, it's easy to get lost in technical jargon and complex solutions.
However, I have found that by focusing on a few key areas and asking targeted questions, you can identify a firm that not only understands the intricacies of the technology but also aligns with your business goals.
Here's a breakdown of what I consider to be the essential questions to help guide your decision-making process.
Balancing Skills
What to Consider: Data scientists are brilliant with statistics and forecasting, but generative AI projects often need more emphasis to be placed on traditional app development skills , particularly as these solutions are often working with documents and process automation, and much less so on numbers. From time to time I’ve seen data scientists engaged unnecessarily to build apps containing generative AI, so it’s worth keeping this in mind.
What to Ask: How does your team combine data science know-how with app development skills? Can you give examples of past projects where both were needed and used effectively? For each role on the project help me understand what value they bring.
Development Stages:
What to Consider: It’s important to know how your project will move through development, testing, and production, especially as test data used to build search indexes may not be the complete corpus of information you may wish to draw from.
What to Ask: Can you walk me through how you handle development, testing, and production environments? What steps do you take to ensure quality at each stage?
Training and Testing Prompts:
What to Consider: You want the AI to respond well, so effective prompt instructions and testing are key.
What to Ask: How do you train and test prompts? Do you use AI to evaluate responses? Can you give examples of how this works for chatbots and other tasks like document processing?
Model Cost and Performance:
What to Consider: Balancing cost and performance is crucial. You want the best bang for your buck. My recommended approach is to start with the most capable model (often most expensive), and then once proven as working, test the solution with less expensive models to see how they perform if it looks like the run costs are significant enough to spend time optimising.
What to Ask: How do you figure out which AI models give the best performance for the price? Once a model is proven, how do you optimize costs?
Solution Architecture:
What to Consider: Make sure every part of the architecture is necessary and justified. There’s a lot of hype around and often I find analytics vendors are recommended to provide a gateway to the LLM, often un-necessarily. This appears to more frequently come up for companies that have experience building “traditional AI” solutions such as machine learning, more often than those firms who have traditionally specialised in app development.
What to Ask: Can you explain why each part of your proposed architecture is needed? If you’re suggesting an analytics platform needs to be included in the design, why is it essential for us?
Low Code vs. Full Code:
What to Consider: Different projects need different approaches, and you want one that fits your needs, balancing capability, supportability and time to go-live.
What to Ask: How do you decide if a low code or full code solution is right for us? What are the pros and cons of each in our case?
Continual Improvement & Tuning:
What to Consider: Feedback is vital for making generative AI solutions better over time, especially with chatbots, and also other AI apps to a varying extent.
What to Ask: How do you collect and use user feedback to improve chatbot responses? What metrics do you track to see how well the chatbot is performing?
Testing Chatbots:
What to Consider: You want testing to match your success criteria. In the case of chatbots, it is often that once launched users will ask questions that may have not been considered or optimised for during the solution design.
What to Ask: How do you make sure your testing aligns with what success looks like for us? Can you show examples of the prompts and test cases you’ll use?
Updating Search Indexes:
What to Consider: Keeping the search index updated is important in an application that relies heavily on search (“RAG architecture”).
What to Ask: How do you handle updates to the search index when content changes? How do you decide what content gets indexed? Is there a method to automate this re-indexing, and delete content that is no longer relevant or out of date?
Managing Permissions:
What to Consider: Protecting sensitive content is non-negotiable in most situations.
What to Ask: How do you manage permissions to ensure only authorized users can access sensitive content? What security measures are in place to protect data?
By asking these kinds of questions, you can really dig into whether a firm is the right fit for your project and ensure they're equipped to deliver what you need.