add tag
4 years ago joulev

This returns errors.

image.png

Here it is very interesting:

  • Space has category code 9 (ignored) (as expected after \ExplSyntaxOn). However the output still has spaces. I expect it to be “11,11,9.” and not “11,␣11,␣9.”, because no ~ is provided.
  • _ and : have category code 11 (letter) (still as expected after \ExplSyntaxOn). So \prg_do_nothing: should be a valid control sequence and it should work. It didn’t, and the error messages implies _ is still considered “math subscript” character (catcode 8).

It is even more interesting when I put the whole \foreach inside explosive syntax:

image.png

It works perfectly! Also bye-bye the spaces between the numbers.

This means either expl3 or pgffor has a bug. Or I really misunderstood something trivial here.

What is happening?

Top Answer
4 years ago Phelype Oleinik

No bug, it’s all working correctly.

In your first example, with \ExplSyntaxOn inside \foreach, the tokens in the loop body have their catcodes frozen, so spaces are tokenised as spaces, and \prg_do_nothing: is tokenised as \prg _do_nothing: (control sequence \prg + two subscript tokens and a bunch of letters).

What seems to have thrown you off tracks is the output of \the\catcode`\_: the expansion of \the\catcode will return the current catcode of that character, and not the catcode of that token. Maybe this helps you see more clearly:

test.png

The example above shows that \the\catcode`\_ outputs the current catcode of \_, but \meaning_ always outputs subscript character _ because that token is frozen.

In your code, even if \ExplSyntaxOn changes the catcode inside the loop body, the control sequence \prg_do_nothing: is already tokenised incorrectly (not as a control sequence).

Your second example then works correctly because the catcodes are changed before TeX sees the tokens, so \prg_do_nothing: is tokenised correctly as a single control sequence. In fact, you could even do:

and the output would be 8,12,10.

4 years
Phelype Oleinik replying to joulev — Saturday, 26th Jun 2021 15:58

TeX eagerly freezes catcodes as soon as it “looks” at a character (either by executing them, by storing them in a macro, or just by passing them as argument to another macro), and since loops in TeX usually require the loop body to be stored in a macro (as \foreach does), I dare say it is not possible to have a loop without prior tokenisation

12 minutes
joulev — Saturday, 26th Jun 2021 15:46

I am kinda surprised to know that the catcode in the loop is frozen though.

joulev — Saturday, 26th Jun 2021 15:41

@Phelype, re: your answer, Thanks for the answer!

an hour
joulev — Saturday, 26th Jun 2021 14:23

I encountered this problem when writing the code for this answer.

Enter question or answer id or url (and optionally further answer ids/urls from the same question) from

Separate each id/url with a space. No need to list your own answers; they will be imported automatically.