BJSS is proud to launch a new virtual assistant designed to bring key insights from our celebrated delivery methodology, Enterprise Agile®. The Enterprise Agile AI assistant marks an exciting step forward in sharing 30 years’ of experience delivering complex software programmes at enterprise scale. The assistant (or chatbot) offers a new way to explore a wealth of advice and insight, demonstrating the possibilities of AI in the process.

Enterprise Agile (EA) is an agile project management approach tailored to tackle challenges that enterprise-level organisations may face. EA provides practical guidance on how to embrace new working methods and technologies that are proven to increase agility while helping organisations remain efficient.

Our AI chatbot is configured to surface insights from the BJSS Enterprise Agile e-book, published in 2023, tapping into a wealth of in-house resources and knowledge. The chatbot was built using Microsoft Copilot Studio, a tool for creating custom bots with cutting-edge AI features. The bot uses generative AI to give sophisticated responses, using the same large language model (LLM) technology as that of Bing Copilot and Chat-GPT.

With insights from the BJSS team, we share the steps to making the development of this virtual assistant a success and how you can start your own GenAI story.

What we learnt over years of developing AI solutions that informed the developmentof our Enterprise Agile AI assistant

“While many conventional approaches to the Software Development Life Cycle (SDLC) still apply to AI-powered software development, several aspects are fundamentally different. Testing cannot rely solely on binary validation (yes or no). Instead, it often involves innovative methods to assess model performance and accuracy.

Bugs are not fixed through straightforward code changes but by exploratory prompt engineering, seeking to steer the AI model's behaviour and leverage its capabilities. This necessitates adaptations in the delivery approach and careful, transparent expectation management with stakeholders. We must convey that these new, sophisticated capabilities introduce an element of unpredictability, due to the non-deterministic nature of AI.”

- Jonathan Hadley, Group Capability Lead for Delivery Management, BJSS

Set clear objectives

Central to our project was the outcome – providing colleagues and clients with an easy way to digest the contents of Enterprise Agile®. The clear objective set from the start allowed us to design and deliver a virtual assistant with a distinct purpose - providing website visitors with answers to questions on the EA methodology. By focusing the assistant on a single source of BJSS content (EA), we minimised potential complexity and risk, leading to swift delivery of a solution.

Assess the scope of the technology

Recognise the capabilities and the limits of the technology available to you. This project leveraged Copilot Studio, which was a beta preview at the time we developed the chatbot. Copilot Studio offers a quick, convenient path to a relatively simple AI chatbot solution. However, working with a new tool that is undergoing rapid development had its challenges, as we needed to adapt to changes.

Some AI projects will require a greater degree of technical control. Depending on your requirements, the choice between using pre-built solutions like Copilot Studio or utilising platforms like Azure AI Studio, Google Vertex AI Studio, Amazon Bedrock or Amazon SageMaker JumpStart should be carefully evaluated. The decision will be based on the factors that are important to your project, such as speed of development, control and risk.

Achieve innovation through iteration

Working with AI tools can be a very different experience to traditional software development, so it's important to leverage the flexibility of iterative methodologies.

Flexible expectations are a must. Not only does AI have elements of unpredictability by nature, but the technology is evolving rapidly around us. The work of an engineer looks different too, often involving considerable exploratory testing while optimising the AI's behaviour. In terms of the testing required, it is in no way traditional. Typical software development lifecycle practices cannot be adhrered to strictly when working on AI-driven projects. For example, we experimented with automated test approaches to assess the suitability of the bot's behaviour and alert us to notable changes. This is a major challenge when the bot's responses are inherently variable.

Ensure your solution is grounded by accurate and reliable sources

The chatbot will serve as a representative of your organisation, so it is crucial to be confident in its responses. This requires careful configuration and prompt engineering. Subtle changes in the AI model's instructions can result in big differences to its behaviour. For certain topics, we opted to set stock responses, rather than leaving it to AI. Before the bot can return a GenAI response, it must rule out the stock responses, search Enterprise Agile for relevant content and apply strict content moderation. This keeps its behaviour on track and mitigates risks.

When dealing with AI, it's crucial to mitigate potential security threats. Although our bot's use case has relatively low risks, we still considered the new set of risks that come with AI (e.g. the OWASP top 10 for LLMs) and the mitigations available to us. We found that Copilot Studio simplified things in this regard - a potential advantage for organisations just beginning to experiment with AI tools.

We found that the content moderation built into Microsoft's Copilot Studio product is strong, even though the product is in its infancy. This bodes well for the future, as we expect safety and security measures to only improve with these kinds of industry-leading products. We found it easy to integrate the bot with our company's Single Sign On (SSO), making our test environments securely accessible for our staff.

Ensuring the appropriate use of AI chatbots is crucial to prevent misinformation, maintain customer trust, uphold company reputation and avoid legal liabilities. There are endless examples of AI tools going off the rails, which is embarrassing at best, extremely costly at worst.

It's critical to use the right kinds of tech and tools in the right contexts. When introducing AI into business processes, it is important to find the balance between capability and predictability. That balance might shift, depending on factors such as the sensitivity of data, the extent of public exposure and the criticality of the business operations involved.

Conclusion

It is vital to set clear objectives, understand the scope and limitations of the driving technology, and to adapt delivery methodologies to meet the unique challenges of AI development.

This AI assistant allows us to actively demonstrate the strong AI expertise held within the team. It provides a novel way to promote the latest BJSS Enterprise Agile® edition and helps to expand our reach and engagement. Additionally, we can actively utilise and train team members in this emerging technology.

Make sure to check out the chatbot and talk to us about how we can support your organisation in leveraging AI tools of your own. Contact BJSS today.