Most optimal approach for text replacement - TeX

TopAnswers TeX

Top Answer

AnonymousRabbit

The functions provided by `etl` to allow expandable replacement might be faster than their unexpandable counterparts of `tl`. However there are limitations (which are also the reason why they are faster):

- `\etl_replace_all_deep:nnn` can't distinguish some tokens (see the documentation of `etl`), and the input token list must not contain `\s__etl_stop` or `\__etl_act_result:n`
- functions defined with `\etl_new_replace_all:Nn` have
	- a fixed search text, so you'll clutter the hash table if you need many
    - mustn't get `\s__etl_stop` as part of the input token list
    
However, as I said already they might be faster (though less versatile) than the unexpandable `expl3` counterparts. Here's a comparison of using `\tl_replace_all:Nnn`, `\regex_replace_all:nnN`, `\etl_replace_all_deep:nnn`, and functions using `\etl_new_replace_all:Nn`:

```#
\documentclass{article}

\usepackage{etl, l3benchmark, booktabs, siunitx}

\ExplSyntaxOn
\tl_new:N \l_my_tmp_tl
\cs_new_protected:Npn \my_replace_text:n #1
  {
    \tl_set:Nn \l_my_tmp_tl {#1}
    \tl_replace_all:Nnn \l_my_tmp_tl { abc } { xyz }
    \tl_replace_all:Nnn \l_my_tmp_tl { lmn } { pqr }
    \tl_replace_all:Nnn \l_my_tmp_tl { a }   { j }
    \tl_use:N \l_my_tmp_tl
  }
\NewDocumentCommand \replace { +m }
  { \my_replace_text:n { #1 } }

\etl_new_replace_all:Nn \__my_exp_replace_abc:nn { abc }
\etl_new_replace_all:Nn \__my_exp_replace_lmn:nn { lmn }
\etl_new_replace_all:Nn \__my_exp_replace_a:nn   { a }
\cs_generate_variant:Nn \__my_exp_replace_a:nn { e }
\cs_generate_variant:Nn \__my_exp_replace_lmn:nn { e }
\cs_new:Npn \my_exp_replace_text:n #1
  {
    \__my_exp_replace_a:en
      {
        \__my_exp_replace_lmn:en
          { \__my_exp_replace_abc:nn {#1} { xyz } }
          { pgr }
      }
      { j }
  }
\NewExpandableDocumentCommand \Replace { +m }
  { \my_exp_replace_text:n {#1} }

\cs_new_protected:Npn \my_replace_text_deep:n #1
  {
    \tl_set:Nn \l_my_tmp_tl {#1}
    \regex_replace_all:nnN { abc } { xyz } \l_my_tmp_tl
    \regex_replace_all:nnN { lmn } { pqr } \l_my_tmp_tl
    \regex_replace_all:nnN { a }   { j }   \l_my_tmp_tl
    \tl_use:N \l_my_tmp_tl
  }
\NewDocumentCommand \replaceall { +m }
  { \my_replace_text_deep:n {#1} }

\cs_generate_variant:Nn \etl_replace_all_deep:nnn { e }
\cs_new:Npn \my_exp_replace_text_deep:n #1
  {
    \etl_replace_all_deep:enn
      {
        \etl_replace_all_deep:enn
          { \etl_replace_all_deep:nnn {#1} { abc } { xyz } }
          { lmn }
          { pgr }
      }
      { a }
      { j }
  }
\NewExpandableDocumentCommand \Replaceall { +m }
  { \my_exp_replace_text_deep:n {#1} }

\cs_new_protected:Npn \my_benchmark:N #1
  {
    \texttt { \token_to_str:N #1 } &
    #1 { abclmnxax }
    \benchmark_silent:n { \hbox_set:Nn \l_tmpa_box { #1 { abclmnxax } } } &
    \fp_use:N \g_benchmark_ops_fp
    \\
  }

\begin{document}
\begin{tabular}{llS[table-format=4.3,round-precision=3,round-mode=places]}
  \toprule
  Macro & Result & {Time~ in~ \texttt{ops}} \\
  \midrule
  \my_benchmark:N \my_replace_text:n
  \my_benchmark:N \my_exp_replace_text:n
  \my_benchmark:N \my_replace_text_deep:n
  \my_benchmark:N \my_exp_replace_text_deep:n
  \bottomrule
\end{tabular}
\end{document}
```

Note that the two macros with `_deep` in their name can replace things inside of braced groups (unlike the other two), and their search text can contain braces and macro parameter tokens (unlike the other two).

![2023-03-08_11h56_56.png](/image?hash=9b1ab5374bf9c2c1162b0323f1dd16a3b505a92c2fc3ac0207a42fdbab815831)

Some strings will have to be replaced first like in my example I would first need to do `abc`->`xyz`, else the last `a`->`j` will turn `abc` into `jbc`. So I suppose this cannot be called a mutually exclusive replacement, right?

Here are two variants of this, pick whichever you like more (I've benchmarked them a little and they seem more or less equally fast): ``` \documentclass{article} \usepackage{etl} \ExplSyntaxOn \NewExpandableDocumentCommand \replaced { +m } { \niranjan_replaced:n {#1} } \cs_new:Npn \niranjan_replaced:n #1 { \exp_not:e { \__niranjan_replaced_loop:nnnN {#1} { abc } { ABC } \__niranjan_replaced_loop:nnnN { def } { DEF } \__niranjan_replaced_loop:nnnN { BCD } { hih } \__niranjan_replaced_loop:nnnN { fgh } { hah } \__niranjan_replaced_loop:nnnN { ghi } { GHI } \__niranjan_replaced_output:n } } \cs_new:Npn \__niranjan_replaced_loop:nnnN #1#2#3#4 { \exp_args:Ne #4 { \etl_replace_all_deep:nnn {#1} {#2} {#3} } } \cs_new_eq:NN \__niranjan_replaced_output:n \exp_not:n \seq_const_from_clist:Nn \c__niranjan_replacements_seq { { abc } { ABC } , { def } { DEF } , { BCD } { hih } , { fgh } { hah } , { ghi } { GHI } } \cs_new_protected:Npn \niranjan_replace_in:N #1 { \seq_map_inline:Nn \c__niranjan_replacements_seq { \tl_set:Nx #1 { \exp_args:No \etl_replace_all_deep:nnn {#1} ##1 } } } \NewDocumentCommand \setreplaced { m +m } { \tl_set:Nn #1 {#2} \niranjan_replace_in:N #1 } \cs_new:Npn \niranjan_benchmark_setreplaced:n #1 { \tl_set:Nn \l_tmpa_tl {#1} \niranjan_replace_in:N \l_tmpa_tl } \cs_new:Npn \niranjan_benchmark_replaced:n #1 { \tl_set:Nx \l_tmpa_tl { \niranjan_replaced:n {#1} } } \ExplSyntaxOff \begin{document} \replaced{abcdefabCDefdefghi{fgh}} \setreplaced\myfoo{abcdefabCDefdefghi{fgh}}\myfoo \end{document} ```

Thanks, I went through the article. I get it now. From the 4 macros compared in the answer I am going with the fourth one (i.e., `\etl_replace_all_deep`). I find it sufficient for my need, but I need to have 29 replacements of this sort to be precise. Shall I ask a new question regarding the performance-comparison of the loop and separate macros?

Loop unrolling means that while in your source code there is a loop the compiled code has instead no loop but a chain of instructions (Wikipedia has an article about it "Loop unrolling"). I'm not sure how I'd create performant code with a sequence to wrap the etl-calls. I'm currently on my mobile, maybe the other rabbit can chime in later today.

Every loop structure that isn't unrolled will add some overhead during runtime. The question is how big that overhead is compared to the loop body's run time. In general you could either store your replacement rules as a `seq` you loop over (`\seq_const_from_clist:Nn \c_niranjan_replacements_seq { { abc } { replace }, { def } { DEF } }` and then loop `\seq_map_inline:Nn \c_niranjan_replacements_seq { \__niranjan_replace:nn #1 }`) or semantically nicer but slower as `key=val` (but not as `prop` list, those stringify their keys).

``` \documentclass[border=1cm]{standalone} \usepackage{etl} \ExplSyntaxOn \cs_generate_variant:Nn \etl_replace_all_deep:nnn { e } \NewDocumentCommand \etlreplace { +m }{ \etl_replace_all_deep:enn { \etl_replace_all_deep:enn { \etl_replace_all_deep:nnn {#1} { abc } { xyz } } { lmn } { pqr } } { a } { j } } \ExplSyntaxOff \begin{document} \etlreplace{{abc}lmnxax} \end{document} ``` This is my code. Is there any way to make it a bit more elegant? I initially had it like `\tl_replace_all_Nnn` (thrice, separate macro for every replacement), but it turned out that as many times I enter that command, the replacement happened those many times and the result was very strange. If I want to have, say 50 such replacements, it would be tedious to loop all of them like this, won't it? So how to neatly organize such multiple replacements? I hope it doesn't reduce the performance.

I am just trying to extract out the usage of all the macros. So basically taking them out from the comparison-macros you have in your answer and using all of them in actual documents like DocumentCommands™, but the expl3 syntax always stumps me.

I meant brace-groups (so `{hidden}` can't be replaced by two of them). Also keep in mind that the `\my_replace_text_deep:n` is not functionally identical to `\my_exp_replace_text_deep:n` (the latter is not using regular expressions, so the overhead in ops might be negligible if you need the extra might of those).

Also, a very basic question since I am not very confident with the expl3 part yet. From your answer it seems that `\etl_new_replace_all:Nn` is the fastest and `\etl_replace_all_deep:nnn` is almost 10+ times slower than it. So, I would prefer using the former, but can you explain in simple words what are the drawbacks of it when compared to the `tl` variant? You say it's not much different, but if we look at the result it is three times faster.

If I understand your code correctly, the first and third (from the table) are the `tl` macros and second and fourth are `etl` macros, right? So with what you have said in this comment, the former should not be able to replace the text inside groups, but it does. If I replace the text provided on line no. 66 to `{ abc\begingroup lmn\endgroup xax }` it still gives the correct output in all the four rows. Why? Is my understanding of a 'group' is wrong?

@निरंजन, re: [your question](#question), also note that the conventional name for a token list variable ends in `_tl` not in `_list`, so `\tl_new:N \l_my_tmp_tl` (`my` being the adhoc module name, `tmp` the variable descriptor, `tl` the type)

@निरंजन, re: [your question](#question), your code won't replace text hidden in groups, that's only possible with either `\regex_replace_all:nnN` or the comparably slow `\etl_replace_all_deep:nnn` of the `etl` package. That being said, you won't get a robust token list replacement function much faster than `\tl_replace_all:Nnn` (iirc. `etl` supports creating new `\tl_replace_all:Nnn`-like functions that are a bit faster, but not much)

1 Answer