pgffor and expl3 conflict - TeX

TopAnswers TeX

Meta

Databases

TeX

Code Golf

APL

C++

.net

db<>fiddle

Java

*nix

PHP

PowerShell

Python

Rust

टेक्-मराठी

Typst

Web Client Dev

Web Server Dev

pgffor and expl3 conflict

add tag

joulev

This returns errors.

```
\documentclass{article}
\usepackage{pgffor}
\usepackage{expl3}
\begin{document}
\foreach \i in {0} {
  \ExplSyntaxOn
  \the\catcode`_, \the\catcode`:, \the\catcode`\ .
  \prg_do_nothing:
  \ExplSyntaxOff
}
\end{document}

```

![image.png](/image?hash=03cd02a78b835c714d8650de0ccb346bcb313b133409e2f8541a0bf30d803b21)

```log
/home/joulev/dev/tmp/tmp.tex:10: Undefined control sequence.
\pgffor@body ...tcode `:, \the \catcode `\ . \prg 
                                                  _do_nothing: \ExplSyntaxOff 
l.10 }

/home/joulev/dev/tmp/tmp.tex:10: Missing $ inserted.
<inserted text> 
                $
l.10 }

/home/joulev/dev/tmp/tmp.tex:10: Missing $ inserted.
<inserted text> 
                $
l.10 }
```

Here it is very interesting:

* Space has category code 9 (ignored) (as expected after `\ExplSyntaxOn`). However the output still has spaces. I expect it to be "11,11,9." and not "11,␣11,␣9.", because no `~` is provided.
* `_` and `:` have category code 11 (letter) (still as expected after `\ExplSyntaxOn`). So `\prg_do_nothing:` should be a valid control sequence and it should work. It didn't, and the error messages implies `_` is still considered "math subscript" character (catcode 8).

It is even more interesting when I put the whole `\foreach` inside explosive syntax:

```
\documentclass{article}
\usepackage{pgffor}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\foreach \i in {0} {
  \the\catcode`_, \the\catcode`:, \the\catcode`\ .
  \prg_do_nothing:
}
\ExplSyntaxOff
\end{document}
```

![image.png](/image?hash=1105abf8125466efb2509814b9a671c782d0cb7a6ac60110966e14b77223d515)

It works perfectly! Also bye-bye the spaces between the numbers.

This means either `expl3` or `pgffor` has a bug. Or I really misunderstood something trivial here.

What is happening?

Top Answer

Phelype Oleinik

No bug, it's all working correctly.

In your first example, with `\ExplSyntaxOn` inside `\foreach`, the tokens in the loop body have their catcodes frozen, so spaces are tokenised as spaces, and `\prg_do_nothing:` is tokenised as `\prg _do_nothing:` (control sequence `\prg` + two subscript tokens and a bunch of letters).

What seems to have thrown you off tracks is the output of ``\the\catcode`\_``:  the expansion of `\the\catcode` will return the _current_ catcode of that character, and not the catcode of that _token_.  Maybe this helps you see more clearly:
```tex
\def\x{cc: \the\catcode`\_\par \meaning_\par}
\tt
\catcode`\_=8  \x
\catcode`\_=11 \x
\catcode`\_=0  \x
\bye
```

> ![test.png](/image?hash=e107a15635faad946c84ab3184e9e8840b1a31cfaf5a100c89b33d6fe6903590)

The example above shows that ``\the\catcode`\_`` outputs the current catcode of `\_`, but `\meaning_` always outputs `subscript character _` because that token is frozen.

In your code, even if `\ExplSyntaxOn` changes the catcode inside the loop body, the control sequence `\prg_do_nothing:` is already tokenised incorrectly (not as a control sequence).

Your second example then works correctly because the catcodes are changed _before_ TeX sees the tokens, so `\prg_do_nothing:` is tokenised correctly as a single control sequence.  In fact, you could even do:
```latex
\ExplSyntaxOn
\foreach \i in {0} {
  \ExplSyntaxOff
  \the\catcode`_, \the\catcode`:, \the\catcode`\ .
  \prg_do_nothing:
}
\ExplSyntaxOff
```
and the output would be `8,12,10.`

1 Answer