Is the doc bot docs, or not?

Upgrading my Shopify email noti­fi­ca­tion templates this morning, I asked Shopify’s LLM-powered devel­oper doc­u­men­ta­tion bot this ques­tion:

What’s the syntax, in Liquid, to detect whether an order in an email noti­fi­ca­tion con­tains items that will be ful­filled through Shopify Collective?

I’d done a few tra­di­tional searches, but couldn’t find an answer, so I thought, okay, let’s try the doc bot!

The reply came, quick and clean:

{% if order.tags contains 'Shopify Collective' %}
  
  
Some items in your order are being fulfilled by a partner merchant through Shopify Collective.
{% endif %}

Looks great. I added this code, but it didn’t work; the con­di­tion wasn’t being satisfied, even when orders had the nec­es­sary tag, vis­ible on the admin screen.

Shopify doesn’t pro­vide a way to test uncon­ven­tional email for­mats without actu­ally placing real orders, so I did my cus­tomary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days.

Turns out, the Shopify Collective tag isn’t present on the order at the time the con­fir­ma­tion email is generated. It’s added later, pos­sibly just moments later, by some other cryptic Shopify process.

I don’t think this is doc­u­mented anywhere, so in that sense, it’s not sur­prising the doc­u­men­ta­tion bot got it wrong. My ques­tion is: what’s the point of a doc bot that just takes a guess?

More point­edly: is the doc bot docs, or not?

And very point­edly indeed: what are we even doing here??

I’ve sent other queries to the doc bot and received helpful replies; I’d cat­e­go­rize these basi­cally as search: “How do I do X?” “What’s the syntax for Y?” But I believe this is a sit­u­a­tion in which the cost of bad advice out­weighs the ben­efit of quick help by 10X, at least. I can, in fact, figure out how to do X using the real docs. Only the doc bot can make things up.

If it was Claude making this kind of mistake, I’d be annoyed but not surprised. But this is Shopify’s sanc­tioned helper! It waits twin­kling in the header of every page of the dev site. I suppose there are domains in which just taking a guess is okay; is the offi­cial doc­u­men­ta­tion one of them?

I vote no, and I think a freestyling doc bot under­mines the effort and care of the folks at Shopify taking the time to write doc­u­men­ta­tion that is thor­ough and accurate.

P.S. One working answer to my ques­tion looks like this:

{% assign has_collective_item = false %}
{% for line_item in line_items %}
  {% if line_item.product.tags contains 'Shopify Collective' %}
    {% assign has_collective_item = true %}
  {% endif %}
{% endfor %}