Creating AgileGPT, an Agile Product Coach, using OpenAI's GPT tools


Over the past two years, I have observed OpenAI's impressive yet sometimes chaotic progress. My interest was piqued with the launch of their GPT-3.5 model and the ChatGPT mobile app in November 2022. I decided to create this blog to capture my experiences with ChatGPT, how it led to creating AgileGPT, and where the journey may lead me. Like many, I downloaded the app and began exploring its capabilities, initially asking for basic factual information. The AI performed admirably in this regard. Naturally, I progressed to more contentious topics like politics and religion, attempting to provoke it into taking firm stances. Surprisingly, it consistently provided balanced views on these controversial issues.

However, as I delved deeper into specific domains, I encountered the limits of its knowledge, leading to less coherent responses and occasional inaccuracies, or what are referred to as "hallucinations." Unfortunately, these erroneous responses were delivered as assertively as the correct ones. So, if you’re not familiar with the domain you’re asking ChatGPT about, how can you determine what is true? Well, I guess AI has truly arrived if it is manifesting the Dunning-Kruger effect! Such human-like characteristics are common when interacting with AI based on Large Language Models (LLMs), which hallucinate due to their reliance on pattern recognition without true comprehension. This results in generating plausible but often incorrect or nonsensical content. While more recent advances in Reinforcement Learning from Human Feedback (RLHF) are improving response accuracy and reducing hallucinations, the industry still has a ways to go. Fortunately, the March 2023 release of GPT-4.0 demonstrated improvements in understanding, contextual awareness, and accuracy, alongside a reduction in these hallucinations.

Despite these advancements, I still encountered frustrating limits of GPT-4.0's knowledge when exploring specific topics in depth. I became curious about enhancing LLM performance through fine-tuning, embedding, and prompt engineering. As I researched OpenAI's APIs and other models like Lhama and Mistral, OpenAI introduced the capability to create custom GPT models in November 2023. With this product, referred to simply as a GPT, you can create a custom version of ChatGPT tailored for a specific purpose with no coding. I have a fair amount of no-code experience from my years building software development tools at Easel, Vigor, and Rational Software. These tools are really good at rapidly completing 80% to 90% of the work, but often it can be nearly impossible to complete the last 10% to 20% of the solution.

To better understand the capabilities and test the limits of these LLM-based AIs, I decided to create my own agent. I picked a domain I knew well - Agile Product Management. I aimed to build an AI coach for scaling Product Management in an agile organization. My original intent was to build out my AI using Python, weaving together a solution based on a combination of OpenAI’s APIs, PyTorch, LangChain, and whatever else made sense. However, I decided to explore prototyping first and use custom GPTs to build the first version of AgileGPT.

While using ChatGPT is free, to create or use a custom GPT, you will need to purchase a premium subscription (Plus, Team, or Enterprise), which starts at $20 per month. For more details, visit https://openai.com/chatgpt.

OpenAI’s GPT Builder allows you to create your custom GPT through an interactive dialogue. You provide information about how you want your GPT to behave and specific data about your domain, and behind the scenes, it generates the instructions that will be used to create the custom experience for your GPT users. If this sounds too good to be true, it is. It will take several iterations to get these instructions right. As you gain experience with creating GPT instructions and LLM prompting, you will likely shift from interactive instruction creation to using the “Configure” feature, where you can directly edit the instructions in a text editor. The instructions created by the GPT builder will be there and can serve as a good starting point. I found creating GPT instructions analogous to creating system prompts using LangChain with the OpenAI APIs. A word of caution: once you start manually editing these instructions, ensure you save them outside of GPT Builder. I found that if you go back and edit in GPT Builder, it overwrites my manual edits. I ended up backing up all my GPT parameters and settings manually in a Google Doc.

While instructions allow you to customize your GPT’s responses in terms of tone, focus, and complexity, you will ultimately be limited by the data on which its model has been trained. That's where the GPT knowledge base feature comes into play. The GPT knowledge base is a feature in the GPT editor where you can upload documents containing specific information about your custom GPT’s domain. This allows you to expand the AI's understanding beyond its pre-existing training data. GPT will dynamically integrate information from these documents into its output during the current session. For AgileGPT, I assembled a combination of public documents and original works for the knowledge base. There are limits to the quantity and size of documents you are allowed to upload. I’ve linked what I believe to be the latest information on file upload limitations in the useful links section at the end of this post.

This is where I started to run into challenges. A physics refresher: the concept of inertia in physics is closely linked to mass. It’s the property of an object that resists changes in its state of motion or rest. Essentially, the greater the mass of an object, the greater its inertia, meaning more force is required to change its motion. This principle is not only integral in classical mechanics but also in training and fine-tuning LLM-based AIs. In LLM training, data can be thought of as the mass of a model that you must overcome to alter its behavior. Just as a physical object with greater mass has more inertia and requires more energy to change its state, so too does your knowledge base need to be substantial to effect a material change in ChatGPT’s GPT-4.0 model. I don’t expect or believe that curating and reinforcing representative data for scaling Agile Product Management was a priority for OpenAI in creating the GPT-4.0 model. As a result, I found that bending and influencing AgileGPT to reflect a deeper understanding of the target domain required a significant effort, including creating the right combination of instructions and knowledge base documents.

Creating AgileGPT wasn’t quite a no-code experience for me. While I had some documents for GPT’s knowledge base that I could upload in their raw native format (mostly PDF), I also had documents in Excel, JSON, and Google Docs. Not only would these documents have exceeded OpenAI’s GPT file capacity limits, but I also wanted to clean and annotate them to ensure better ingestion. As a result, I created a semi-automated data pipeline tool in Python to prepare and combine files for upload. This was overkill for the current version of AgileGPT, but I felt the investment was worth it as I envisioned a future version with a more extensive and curated domain-specific model that was more under my control.

One GPT feature I did not exploit was Actions. Actions allow you to enhance GPTs with the ability to access external data such as databases, email systems, or other systems. I may explore future AgileGPT integrations with collaboration and product planning tools using actions.

I did encounter several “product maturity” challenges while building AgileGPT. While not surprising given OpenAI’s rapid innovation, some issues were nonetheless disappointing and frustrating. Using the GPT Editor, I frequently received error messages with no explanation as to their cause or impact, which would then disappear. This was neither informative nor helpful. Many of these errors occurred while uploading documents to the knowledge base, which I managed to get through with brute force and many retries. Or so I hope (read on). When editing and updating AgileGPT, the GPT editor indicates that it is published, but right next to the “Update” button, a persistent message says “Unpublished changes.” There is no feedback on what specific changes are unpublished, making it impossible to determine the impact on users interacting with AgileGPT. Despite trying various purported solutions found in forums, none resolved these errors. After reporting the issue to OpenAI, I was unable to satisfactorily resolve it. In general, the opacity of error handling was very problematic and unacceptable.

I also attempted to use OpenAI Customer Support to resolve these issues. Surprisingly, it was poor, given that OpenAI and LLM technology should be well-suited to disrupt and transform technical support positively. It's unclear if the support chatbot actually uses OpenAI technology, as it has a restrictive menu-based front-end with very limited natural language understanding. Frankly, if this experience is built using OpenAI, I would question the platform's viability. I hope improving this is on OpenAI’s near-term roadmap.

My journey with OpenAI and its evolving technology is far from over. My experiences, from initial experiments with ChatGPT to creating AgileGPT, have been a blend of discovery and challenge, reflective of the broader narrative in AI development. I’ve learned a lot and, despite the challenges, am awestruck and bullish on the potential of Large Language Models. My experiences, ranging from detailed training and fine-tuning of a model to tackling customer support issues, highlight that AI technology, while advanced, is still in a phase of maturation. Looking ahead, I will continue to improve AgileGPT. I plan to explore custom implementations using base models from OpenAI, Lhama, Mistral, and other suitable technologies. Stay tuned for more posts on the progression of AgileGPT.

Some useful links referenced in this post:

Cross-posted to LinkedIn here.



Comments

Post a Comment

Popular posts from this blog

Be agile about agile!

The democrats and republicans are doing a fine job of "fire-bombing" the centrists in both their parties (I grabbed that quote from an article "It's not easy being George" in this month’s Vanity Fair). And, as I scan the agile groups and blogosphere, it seems the methodologists are doing it as well. Some of us take the "extreme" in extreme programming too literally, and others are holding on to some of agile developments long disproved stigma for far too long. My approach always has been (and hopefully always will be, unless of course I turn out like my father, which all of us eventually do...) a pragmatic one. There are many things I love about Scrum and agile - driving elaboration and development off the backlog, test-driven development, burndown charts, and much more. And there are a some that I could do without (self-organizing teams, developers and testers are interchangeable, ... I blog on these later). An agile approach needs to be tailored to y...
Read more

When Do You Need Agile Product Portfolio Management?

Image
First, let’s define Agile Product Portfolio Management. Agile Product Portfolio Management is a dynamic approach to managing a company's suite of products. It involves continuously evaluating and prioritizing products based on shifting market demands, customer feedback, and business goals. It integrates principles of Agile methodology - such as flexibility, rapid iteration, and cross-functional collaboration - into portfolio management. The focus is on adapting quickly to changes, making swift decisions based on current data, and ensuring that resources are allocated efficiently to maximize product success and overall business value. It's about maintaining a balance between long-term strategic vision and short-term market realities, ensuring the product portfolio remains relevant and competitive.  Agile Portfolio Management is needed when your business and product line achieve a size, complexity, and maturity where decisions can no longer be locally optimized for individual pro...
Read more