Conversation
Summary
|
Temporary markers on the product_catalog page to determine which wrappers the RTD/Cloudflare HTML->Markdown converter strips. Will be removed once the experiment concludes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HTML->Markdown converters used by AI agents (e.g. the one behind Read the Docs' 'Accept: text/markdown' content negotiation) skip <nav>/<header>/<footer> elements, but only until the first closing tag - nested <nav> elements make them leak most of the sidebar into the converted Markdown (~170 lines of nav noise per page). Restructure the sidebar so each skippable region is a single <nav> with no nested nav/header/footer inside: - main.html: the main_nav wrapper div becomes the one <nav> landmark - nav.html / nav-item.html: inner md-nav elements become <div>s - toc.html: inner group <nav>s become <div>s; the outer wrapper stays <nav> standalone but renders as <div> when embedded in the sidebar Styling and behavior are unchanged: all CSS targets classes and Material's JS targets data-md-component attributes. Also removes the temporary converter probe markup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The llms.txt spec expects the summary right after the H1 title to be a blockquote. mkdocs-llmstxt emits markdown_description verbatim, so prefix it with '> ' to produce the expected structure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| |`multiple`| true</br>false|Boolean. To allow users to select multiple items.| | ||
| |`translation_domain`|true</br>false|Used for translating choices and placeholder.| | ||
| |`custom_form`|true</br>false|For custom form must be set to true.| | ||
| |`multiple`| true<br>false|Boolean. To allow users to select multiple items.| |
There was a problem hiding this comment.
</br> is replaced with <br> everywhere in this PR - </br> is not a valid tag and some HTML -> Markdown converters are failing to parse it.
Compare:
Current doc: https://doc.ibexa.co/en/5.0/administration/back_office/back_office_elements/add_dropdowns/index.md (truefalse in the output)
This PR: https://ez-systems-developer-documentation--3297.com.readthedocs.build/en/3297/administration/back_office/back_office_elements/add_dropdowns/index.md (true false in the output)
| | `fieldSettings` | array | Settings of the field (depends on the field type) | | ||
| | `parameters` | hash | Options passed to `ibexa_render_field()` under the `'parameters'` key | | ||
| | `attr` | hash | The attributes to add the generate the HTML markup, passed to ibexa_render_field()` under the `'attr'` key. <br> Contains at least a class entry, containing <fieldtypeidentifier>-field | | ||
| | `attr` | hash | The attributes to add the generate the HTML markup, passed to `ibexa_render_field()` under the `'attr'` key. <br> Contains at least a class entry, containing `<fieldtypeidentifier>-field` | |
There was a problem hiding this comment.
Small changes here to the formatting as well -as it was wrong.
| # Pin links in the generated Markdown (and llms.txt) to this branch's | ||
| # version instead of site_url's en/latest, without affecting the HTML | ||
| # site's canonical URLs. Update when branching a new version. | ||
| base_url: https://doc.ibexa.co/en/5.0/ |
There was a problem hiding this comment.
Thanks to this change, links to HTML files point to the correct version instead of latest:
Compare (current):
https://doc.ibexa.co/en/latest/api/rest_api/rest_api_usage/rest_api_usage/index.md
| # The "> " prefix renders the description as a blockquote summary, | ||
| # as expected by the llms.txt spec (https://llmstxt.org) | ||
| markdown_description: >- | ||
| > Developer documentation for Ibexa DXP — architecture, APIs, templating, |
There was a problem hiding this comment.
The spec defines that the description must be specified as a blockquote:
https://afdocs.dev/checks/content-discoverability#how-to-fix-1 , https://llmstxt.org/#format
This PR adds that:
https://ez-systems-developer-documentation--3297.com.readthedocs.build/en/3297/llms.txt
| site_url: https://doc.ibexa.co/en/latest/ | ||
| hooks: | ||
| - hooks/edit_uri.py | ||
| - hooks/eol_status.py |
There was a problem hiding this comment.
Fixing an incorrect merge - we had a double hooks section, the latter overwrote the former without any warning.
| (which is present in the served page whether the visible warning applies | ||
| or not). | ||
| """ | ||
| current_version = os.environ.get("READTHEDOCS_VERSION_NAME", "") |
| </div> | ||
| {% endif %} | ||
| {% include "partials/eol_warning.html" %} | ||
| {% if config.extra.current_version_is_eol %} |
There was a problem hiding this comment.
Only included when needed
| {% endfor %} | ||
|
|
||
| <div class="{{ ns.bootstrap_extra_css }}"> | ||
| <div style="position:absolute;width:1px;height:1px;clip:rect(0 0 0 0);overflow:hidden"> |
There was a problem hiding this comment.
When testing this (screenshot in ibexa/documentation-user#415), it turns out that the agents trim our the head section when requesting the content, and our Markdown version was not acessible.
See https://afdocs.dev/checks/content-discoverability#llms-txt-directive-html and https://afdocs.dev/checks/content-discoverability#llms-txt-directive-md in the specification for guidelines how to fix this.
It's important that the spec directly mentions this:
Avoid display: none, which some converters strip. The directive should be present in server-rendered HTML; avoid relying solely on client-side JavaScript injection, since most agents fetch pages without executing JS.
(https://agentdocsspec.com/spec/#llms-txt-directive-html)
which would be my first approach - but it's not advised.
| def inject_page_metadata( | ||
| content: str, description: str = "", editions: list = (), llms_txt_url: str = "/llms.txt" | ||
| ) -> str: | ||
| """Insert the llms.txt pointer, page description, and an 'Editions: X, Y' line after the first h1 heading. |
There was a problem hiding this comment.
Implements this part of the spec:
https://afdocs.dev/checks/content-discoverability#how-to-fix-6
| {{ nav_item.title }} <span class="pill pill--new" hidden>New</span> | ||
| </label> | ||
| <nav class="md-nav" aria-label="{{ nav_item.title }}" data-md-level="{{ level }}"> | ||
| {# div instead of nav: nested <nav> tags break HTML->Markdown converters, see nav.html #} |
There was a problem hiding this comment.
All the nav > div changes are a result of Fable investigation into the ReadTheDocs HTML -> Markdown converter.
Currently, requesting a Markdown version of the site begins with a long TOC list:
~/Desktop/Sites/v5 main +1 !178 ?30 ❯ curl -H "Accept: text/markdown" https://doc.ibexa.co/en/5.0/administration/back_office/back_office_elements/add_dropdowns/ 24.4.0 14:24:40
---
description: Add custom drop down menus to back office interface.
---
<!doctype html>
[Skip to content ](#add-drop-downs)
[Developer Documentation](https://doc.ibexa.co/en/latest/)
* Getting started New
* Tutorials New
* Page and Form tutorial New
* Generic field type New
* API New
* REST API New
* [REST API reference ](https://doc.ibexa.co/en/5.0/api/rest%5Fapi/rest%5Fapi%5Freference/rest%5Fapi%5Freference.html)
* Extending REST API New
* [REST API authentication ](https://doc.ibexa.co/en/5.0/api/rest%5Fapi/rest%5Fapi%5Fauthentication/)
* GraphQL New
* Event reference New
* [Notification channels ](https://doc.ibexa.co/en/5.0/api/notification%5Fchannels/)
* Administration New
* Dashboard New
* Admin panel New
* Content organization New
...
Fable investigation:
Can the nav/ToC be removed? Yes — done. Probe experiments on the preview
revealed the converter's actual behavior: it skips <nav>/<header>/<footer>
subtrees, but only until the first closing tag — it doesn't handle nesting.
Material's sidebar nests <nav> inside <nav>, so every inner </nav> ended the
skip early and leaked ~170 lines of nav items. (Visibility attributes like
hidden/aria-hidden/display:none are ignored entirely, and <aside>/<div> are
never skipped.)
This change reworks the navigation so that there are no nested nav tags.
After the changes:
~/Desktop/Sites/v5 main +1 !178 ?30 ❯ curl -H "Accept: text/markdown" https://ez-systems-developer-documentation--3297.com.readthedocs.build/en/3297/administration/back_office/back_office_elements/add_dropdowns/
---
description: Add custom drop down menus to back office interface.
---
<!doctype html>
[Skip to content ](#add-drop-downs)
For AI agents: the complete documentation index is available at [llms.txt](https://ez-systems-developer-documentation--3297.com.readthedocs.build/en/3297/llms.txt); this page is also available as markdown at [index.md](https://ez-systems-developer-documentation--3297.com.readthedocs.build/en/3297/administration/back%5Foffice/back%5Foffice%5Felements/add%5Fdropdowns/index.md).
* Documentation >
* Administration >
* Back office >
* Back office elements >
* Add drop-downs
# Add drop-downs[¶](#add-drop-downs "Permanent link")
In Ibexa DXP, you can create a reusable custom drop-down and implement it anywhere in the back office. Follow the steps below to learn how to integrate this component to fit it to your project needs.
## Create `<select>` input[¶](#create-select-input "Permanent link")
The breadcrumbs are still here, but IMHO it's ok for now.
mnocon
left a comment
There was a problem hiding this comment.
Comments for reviewers
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
* AFDocs: Improve docs accessibility for agents * Fix * TEMP: Probe RTD markdown converter element handling Temporary markers on the product_catalog page to determine which wrappers the RTD/Cloudflare HTML->Markdown converter strips. Will be removed once the experiment concludes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TEMP: Round 2 converter probes with link-rich content Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * AFDocs: Keep sidebar nav out of agent-facing Markdown output HTML->Markdown converters used by AI agents (e.g. the one behind Read the Docs' 'Accept: text/markdown' content negotiation) skip <nav>/<header>/<footer> elements, but only until the first closing tag - nested <nav> elements make them leak most of the sidebar into the converted Markdown (~170 lines of nav noise per page). Restructure the sidebar so each skippable region is a single <nav> with no nested nav/header/footer inside: - main.html: the main_nav wrapper div becomes the one <nav> landmark - nav.html / nav-item.html: inner md-nav elements become <div>s - toc.html: inner group <nav>s become <div>s; the outer wrapper stays <nav> standalone but renders as <div> when embedded in the sidebar Styling and behavior are unchanged: all CSS targets classes and Material's JS targets data-md-component attributes. Also removes the temporary converter probe markup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * AFDocs: Render llms.txt description as blockquote summary The llms.txt spec expects the summary right after the H1 title to be a blockquote. mkdocs-llmstxt emits markdown_description verbatim, so prefix it with '> ' to produce the expected structure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update theme/main.html Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
* AFDocs: Improve docs accessibility for agents * Fix * TEMP: Probe RTD markdown converter element handling Temporary markers on the product_catalog page to determine which wrappers the RTD/Cloudflare HTML->Markdown converter strips. Will be removed once the experiment concludes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TEMP: Round 2 converter probes with link-rich content Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * AFDocs: Keep sidebar nav out of agent-facing Markdown output HTML->Markdown converters used by AI agents (e.g. the one behind Read the Docs' 'Accept: text/markdown' content negotiation) skip <nav>/<header>/<footer> elements, but only until the first closing tag - nested <nav> elements make them leak most of the sidebar into the converted Markdown (~170 lines of nav noise per page). Restructure the sidebar so each skippable region is a single <nav> with no nested nav/header/footer inside: - main.html: the main_nav wrapper div becomes the one <nav> landmark - nav.html / nav-item.html: inner md-nav elements become <div>s - toc.html: inner group <nav>s become <div>s; the outer wrapper stays <nav> standalone but renders as <div> when embedded in the sidebar Styling and behavior are unchanged: all CSS targets classes and Material's JS targets data-md-component attributes. Also removes the temporary converter probe markup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * AFDocs: Render llms.txt description as blockquote summary The llms.txt spec expects the summary right after the H1 title to be a blockquote. mkdocs-llmstxt emits markdown_description verbatim, so prefix it with '> ' to produce the expected structure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update theme/main.html Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>

Target: 5.0, 6.0
Companion PR: ibexa/documentation-user#415
This PR improves how AI Agents can read our documentation.
Things done:
Follow Agent-Friendly Documentation spec
AFDocs is a specification for creating docs accessible by AI agents.
The current score (after #3161) is:
This PR aims to improve the first three criteria:
ReadTheDocs Markdown conversion
When a request with
Accept: text/markdownis made, ReadTheDocs serves the Markdown version of the page:https://docs.readthedocs.com/platform/latest/reference/markdown-for-agents.html
This Markdown conversion is done independently from what we do to serve our Markdown, and there are limited ways we can influence it.
But, importantly, there are issues with that conversion that we need to fix/work around.
EOL version warning
Sending a request with
curl -H "Accept: text/markdown" https://doc.ibexa.co/en/5.0/administration/back_office/back_office_elements/add_dropdowns/has the following in the page content:This is of course very bad, and in this PR the solution is reworked to add the EOL warning only when building for specific branches.
Otherwise, AI Agents might think that 5.0 is EOL already.
Current state: add the warning and hide it when not needed by using JS
This PR: do not add the warning unless the branch is marked as EOL.
The EOL version change has to be backported to 4.6 as well
Hiding the TOC in the Markdown
See https://github.com/ibexa/documentation-developer/pull/3297/changes#r3622091055
After the changes
It's hard to get an exact score before this is merged, but it looks much better:
BTW the following fail:
is a false postive: ReadTheDocs returns Markdown, but starts the response with "<!doctype html>" which is treated by the tool as HTML response.