I'm having trouble rolling out document agents uisng ai. Has anyone tried this?

[Mark Bayly]

Hi all,

I've been using ChatGPT to help me write an agent that checks whether an SOP topic exists in our knowledge base and then assists the person in writing the SOP if there is no overlap. I was able to get a set of instructions that worked after a bit of trial and error, but then I tried to roll it out to another agent who would check and create a How-To document with the same sort of format, but tailored to the contents of a How-To.

ChatGPT just can't do it. I then tried Claude and it can't either. I asked Gemini how to sort it out, and it suggested I create a JSON schema to help the AI build the proper instructions. Hours later, and it just doesn't do it. I could have sat there and transcribed it myself and not had all the problems, but it seems like something that should be straightforward. It doesn't appear to be an issue with the Rovo ai.

Has anyone had any luck coming up with a solution to do this?

Any guidance or ideas appreciated.

[Dr Valeri Colon _Connect Centric_]

Yes, you’re not alone—rolling out document agents with tailored instructions is tricky. Even strong LLMs can struggle to adapt instructions across similar formats like SOPs and How-Tos. Success often depends on narrowing scope, simplifying instructions, and anchoring responses to a structured template or schema. If it helps, others have found better results when they test and refine prompts within the same interface before scaling.

Check out this resource on prompting https://github.com/ibuchanan/forge-rovo-metaprompting/tree/main/prompts

[Mark Bayly]

@Dr Valeri Colon _Connect Centric_ thanks for trying to help, but I'm a no-code person, so the information in the link is miles above my head and beyond my understanding. I get the drift, though. Ask Rovo how to do it. I might give it a try, although if it isn't written in my Confluence database, is it going to find anything to tell me?

In my opinion, Rovo appears to be nowhere near ready to deliver the sort of functionality and performance that Atlassian claims for a no-code product. If I launched a product like this in the real world, I'd be sued and fined by the ACCC in Australia for false advertising and a whole lot of other consumer law infringements. No wonder it is free. 

I've been trying to participate in the community, but it seems to be geared towards software engineers, rather than business owners like myself.

Unfortunately, I don't have months of my life to devote full-time to trying to understand how this product works. 

I'm sure I'm not alone, but I'm incredibly frustrated by the lack of Atlassian employees involved in resolving the problems. I sat through a webinar last week that was a waste of time. Asked a question about roadmaps and was told that they couldn't put anything on the roadmap because they had no idea what was going to happen in the future. I was not impressed at all — pure lip service from product and technical people with their heads stuck in technology and no customer view at all.

I'm in my second week of multiple emails daily from senior software support at Atlassian trying to get Rovo connected to my Google Drive for heaven's sake!! Fully integrated NOT.

After many years on Confluence, I think it's time to find a better solution. Maybe Atlassian AI will pick up my sentiment. There is no other channel to be heard and I've given up providing a voice to no-one.

[Dr Valeri Colon _Connect Centric_]

Hi @Mark Bayly—thank you for your honesty. You’re absolutely right that the AI experience can feel overwhelming, especially for business users expecting no-code simplicity. You shouldn’t need to be technical to use AI effectively.

If you don't mind, I'll use your example to create the agent in our demo environment  and make instructions on how to copy/paste and edit the agent for your site. That way you can test it out and determine whether its somewhere you'd like to devote your time.

Here's the approach I'll take: create a sample SOP or How-To directly in a Rovo agent's instructions, and put the same example in a Confluence page (giving the agent 2 reference point). Then, when using the agent, tell Rovo to find and apply the format to the page you select. It’s not perfect, but it avoids code or JSON.

Feedback like yours is vital to getting Rovo where it needs to be. If you would send me the name and date of the webinar at valeri.colon@atlassiancommunity.com it will help improve the experience for other. I can also jump on a call if you're tired of sending emails.

I do not work for Atlassian, I am volunteer in their Community Champion program, a leader in the community-led potion of AI adoption and author of Make AI Work: Cutting Through the Hype to Deliver Real Business Value. And no that's not a plug to go buy the book, I go through the research and real world-based principles in our free community-led classes each month. Here's the video from our last class on Smart Agents for Confluence.