We started by checking /llms.txt on 500 dealership domains. We retrieved a text guide or index on 261 of them, or 52.2%.

Then we followed selected links to see what they supplied.

That second step was more useful. Some inventory feeds provided detailed vehicle records but only part of the declared inventory. Prices could differ between the feed and the vehicle page. Incoming vehicles could still carry an InStock label, and a feed on one dealership’s website could include records naming other stores.

If you’re evaluating an agent or talking with your website provider, those are the details I’d want to check. Finding a file is a starting point. You still need to see where it leads and whether the information answers the question you’re asking.

What we found at llms.txt

Outcome	Domains	Share of sample
Text index or guide retrieved	261	52.2%
Response indicated the file was missing	130	26.0%
Request deferred because usable robots.txt guidance was unavailable	106	21.2%
Successful response with an empty body	1	0.2%
HTML returned at the text-file path	2	0.4%

The 106 deferred requests are unknown results. We didn’t count them as missing files. Our collector stopped further requests when access, server or transport problems prevented it from obtaining usable robots guidance.

We also separated empty responses and HTML from actual text guides, even when the server returned a successful status.

The 52.2% figure applies to this sample. It doesn’t tell us how many American dealerships use AI, appear in AI answers or fully follow the llms.txt proposal.

Results at llms.txt: 261 text indexes or guides, 130 missing, 106 deferred, one empty and two HTML responses across 500 sampled domains.

Most retrieved files were lists of links

Of the 261 text resources, 180 were unsectioned lists of links and descriptions. They commonly pointed to inventory, vehicle details, model information and service pages.

The llms.txt proposal calls for a top-level Markdown title. Our retrieval count was broader than strict compliance with that format.

A link list can still help an agent find a page. But it doesn’t tell us whether the destination has current inventory information, whether the agent can read it without a browser, or whether a commercial AI service uses it.

We also saw shared templates across websites. Those similar files shouldn’t be treated as separate strategic decisions made by every dealership.

The feeds had useful facts, along with some gaps

From 21 domains with explicit AI instructions and structured-inventory links, we inspected 24 selected JSON responses.

Twenty-one contained vehicle records. Three returned empty vehicle arrays and a declared total of zero for the selected categories. That didn’t establish that those dealerships had no inventory elsewhere.

Across the responses, we counted 887 vehicle rows with 887 distinct VIN strings. Every row included a nonempty VIN, stock number, year, make, model, dealer identity and location, and a vehicle-page URL. A separate recount matched the original extraction.

We followed 18 sampled vehicle links. All 18 returned HTML with the matching VIN in the initial body text. That confirmed those connections, not the accuracy of every field or every link in the feeds.

There were also consistent omissions. All 887 rows had imageUrls: null, and the inspected vehicle objects lacked an explicit currency field. The ordinary vehicle pages could supply those details, so these were gaps in the feed representation.

All 24 responses carried X-Robots-Tag: noindex,nofollow. The feed URLs were permitted by the robots rules we evaluated. Fetch permission and indexing directives are separate, and neither observation tells us exactly how every AI service will handle the information.

A successful response may cover only part of the inventory

Fourteen of the 24 JSON responses declared another page. A separate pagination check found additional VINs that weren’t in the first response.

An agent that stops after the first successful request may have a clean-looking list that’s incomplete.

For an inventory comparison, I’d want the report to show how many records it actually inspected, how many the source claimed, and whether it followed the remaining pages. Otherwise, it’s too easy to mistake a partial result for the whole inventory.

Price fields need more than a number

The feeds separated MSRP, selling price, dealer discount and cash rebates. Those labels help, but they don’t automatically explain how the amounts relate.

Every record had dealerDiscount: 0, even though 463 records had different MSRP and selling-price values. Of the 112 records with positive cash rebates, 51 still had equal MSRP and selling-price values.

That doesn’t tell us which amount a particular buyer qualifies for or whether a rebate has already been applied. An agent shouldn’t subtract it again without checking.

Three records had a zero selling price. Follow-up observations included values that changed, persisted or differed from the ordinary vehicle page. We couldn’t establish the cause, and we didn’t interpret them as free-car offers.

We also compared the feed with structured data on 18 matching vehicle pages. In 16 pairs, the page’s structured offer price differed from the feed’s selling price, even though the captures were less than three seconds apart.

Those were not 16 confirmed pricing errors. Five page prices matched the feed’s MSRP, and another difference reconciled with a disclosed fee. Other differences remained unresolved.

Finance and lease disclosures added their own payment amounts and eligibility conditions. The useful answer needs to keep those details with the price.

The vehicle page could add information and still disagree

Each of the 18 matched vehicle pages had a structured vehicle object containing USD currency, an image URL and an InStock assertion.

That filled in some information missing from the feed. It didn’t independently verify the photograph or confirm that the vehicle was ready for delivery.

Mileage differed in four pairs: the feed showed a small nonzero reading while the page’s structured vehicle object showed zero. We verified the difference between the records, not the physical odometer.

When two sources disagree, I’d want the agent to show both values and explain what it checked before choosing one.

Store and transit fields need their own review

Two captured feeds included records naming multiple dealerships while consistently identifying the host website in a separate domain field.

Shared inventory may be intentional. But an agent preparing a store-specific report needs to use the record’s dealership information, rather than assigning every VIN to the website’s store. Neither field proves where the vehicle is physically parked.

We also followed the complete group of 22 records marked inTransit: true. All 22 retained that flag in refreshed feed responses. All 22 corresponding vehicle pages had a structured offer asserting InStock.

Two pages selected beforehand received browser checks. Both visibly said the vehicle was in transit while retaining InStock in the structured data. We didn’t perform rendered-page verification for all 22.

Twenty-one of the transit-flagged records also had an inventory date. One checked page showed a future arrival estimate even though the feed listed an earlier inventory date.

That matters if you’re asking for aged inventory. Subtracting that date from today doesn’t establish how long the vehicle has physically been on the lot.

Some records labeled new also carried a loaner flag. We didn’t verify title history, prior use or warranty start dates. The extra flag simply showed why filtering on “new” alone could leave out a relevant distinction.

The format doesn’t settle whether the information is complete

We inspected selected Markdown resources presented for AI, along with ordinary HTML and structured records.

Matched versions could agree on the VIN and total price while describing a price component differently. Media information also varied.

Some routes returned different representations depending on the request’s Accept header. That showed what our research client received under those conditions; it didn’t establish what every commercial crawler would receive.

We also found that a plain-text extraction could miss vehicle cards even when machine-readable vehicle records were embedded in the HTML.

So the agent’s capabilities matter too. It needs to know what it retrieved, what it parsed and what it couldn’t inspect. A JSON or Markdown response can make information easier to process, but it still needs checking.

“AI allowed” is too broad to be a useful result

We retrieved 385 parseable robots.txt files. For each, we identified the rule group applicable to GPTBot and OAI-SearchBot.

Applicable rule group	GPTBot	OAI-SearchBot
Matching named group	203	221
Wildcard group	182	164
Total parseable files	385	385

These counts describe which groups applied. They aren’t counts of allowed or blocked websites.

We then evaluated 725 selected inventory-purpose destinations, including query strings. Saved policies permitted both bots in 694 cases and excluded both in three. Another 28 couldn’t be assessed. There was no permission difference between the two bots within the 697 assessable cases.

Permission didn’t guarantee a successful request. Twenty-one of the 694 permitted destinations returned HTTP 403 to our research client, even though its requests were also allowed by the saved rules.

That tells us what happened to our client. It doesn’t establish why the server rejected the request or what a commercial crawler received.

A useful access report should keep the bot name, exact URL, applicable rule and observed response together. Training collection, search access and user-triggered visits also serve different purposes, so they shouldn’t be collapsed into one “AI access” label.

Five checks I’d request for your dealership

Follow one vehicle through the available sources. Start with the guide or inventory feed, then compare it with the vehicle page. Show which facts are available in text, structured data and the completed browser view.
Check how much inventory was actually inspected. Follow pagination and separate the records reviewed from the source’s declared total. Confirm which store each record names.
Look at an incoming vehicle. Compare the transit flag, visible arrival estimate, availability label and inventory date before using it in an on-lot or aged-inventory report.
Work through one price comparison. Keep MSRP, selling price, fees, rebates and eligibility conditions attached to their amounts. Explain any difference that remains unresolved.
Test the access route the agent intends to use. Record crawler permissions, indexing directives and the actual response separately.

These checks should leave you with specific questions your team or provider can answer. They also help you judge whether the agent has enough evidence for the report it’s producing.

What this study can and can’t tell us

The sample covered 500 distinct requested domains associated with locations across 40 states, with 125 in each Census region. It was directory-derived and not nationally representative. Multi-location businesses counted once by domain, so this wasn’t a census of 500 rooftops.

The final group included 458 selections from an existing directory and group-roster frame, two same-region replacements and 40 purposefully selected used-vehicle retailers. The replacements addressed placeholder websites, and that eligibility adjustment happened during collection. Missing files, access blocks and network failures remained recorded outcomes.

We selected 2,304 linked URLs across 390 domains. The other 110 had no selected destinations. Each domain had a limit of six initial linked resources, with up to three selected HTML resources receiving paired browser observations. Destination selection prioritized relevant resources and wasn’t random.

Separate follow-ups examined pagination, representation differences, zero prices and the incoming-vehicle group. Those checks didn’t enlarge the 500-domain sample or change its baseline percentages.

Requests identified the client as DealerClawResearchBot and followed its recorded robots permissions. Browser visits used bounded waits and scrolling, with some requests and frames restricted. Those limits matter when interpreting information that didn’t appear.

Automated extraction and AI-assisted review supported the work. There was no independent human review panel.

We retained the identified responses and audit trail privately. Public files contain aggregate results and generalized examples, allowing readers to check reported calculations rather than independently re-crawl the sample.

We didn’t measure AI rankings, citations, traffic, commercial crawler visits or whether adding llms.txt improves those outcomes. What we did find was a practical set of information and access checks worth making before relying on an agent’s inventory answer.

We checked what an agent could actually read across 500 dealership websites