The best answer depends on *exactly* what you want.
As it stands, the question is far too vague. I can offer more than a half-dozen solutions, but whether any of them are “correct” depends on:
* What do you expect to get from a string that *starts* with one or more spaces? Should the first string in the result vector be an empty string?
* What do you expect to get if the string *ends* with one or more spaces? Should there be an empty string at the end?
* What do you expect to get if the string is *entirely* made up of spaces? Should the result vector be empty, or should it contain one empty string, or two?
Also, a lot depends on what kind of usage patterns and performance characteristics you want.
* Is the tokenizing function expensive or cheap? Simple splitting on spaces or a small set of whitespace characters is cheap, but for more advanced tokenizing, like, say tokenizing C or C++ code, that would be more expensive.
* Do you want *views* into the original input? Getting the tokens as sub-views of the input may be useful or even necessary for some cases, but prevents other stuff (like tokenizing a file or network stream directly).
I’m just going to have to guess at what you really want. 🤷🏼
## The easy solution
The easiest solution that seems to satisfy all your requirements is to use [a `chunk_by` view](https://cppreference.com/cpp/ranges/chunk_by_view).
(The reason `views::chunk_by` is superior to `views::split` is because `views::split` only splits on a single character, and one of the issues with the original code is that it “only delimits based on spaces, not other white space”. If we want to delimit by *multiple* characters, not just one, we need to use a function. And to use a function to delimit, we need `views::chunk_by`.)
First we need a function to detect whitespace:
```
constexpr auto is_whitespace(char c) noexcept
{
return c == ' ' or c == '\t';
}
```
You can extend that to detect newlines or whatever else you please.
For `chunk_by` we need a function to detect when *not* to split chunks; a function that is true over every pair of elements *within* a single chunk, but not where two chunks have to be split apart. In our case, within a chunk, all pairs of characters are either both *not* whitespace (within a token), or are whitespace (between tokens):
```
auto const token_chunker = [](auto a, auto b) { return is_whitespace(a) == is_whitespace(b); };
```
After we break the input into chunks, we want to discard the whitespace chunks and keep only the token chunks. So we need another lambda:
```
auto const is_token = [](auto&& chunk) { return (not std::ranges::empty(chunk)) and (not is_whitespace(*std::ranges::begin(chunk))); };
```
So our tokenizer is just:
```
input
| std::views::chunk_by(token_chunker)
| std::views::filter(is_token)
```
This works fine, but the views it produces are arbitrary views of characters. If we’re given an input string (view), we probably want to get the tokens as string views. But that’s no problem:
```
input
| std::views::chunk_by(token_chunker)
| std::views::filter(is_token)
| std::views::transform([](auto&& view) { return std::string_view{view}; })
```
So the solution in its entirety is:
```
constexpr auto is_whitespace(char c) noexcept
{
return c == ' ' or c == '\t';
}
constexpr auto Foo(std::string_view input)
{
auto const token_chunker = [](auto a, auto b) { return is_whitespace(a) == is_whitespace(b); };
auto const is_token = [](auto&& chunk) { return (not std::ranges::empty(chunk)) and (not is_whitespace(*std::ranges::begin(chunk))); };
return input
| std::views::chunk_by(token_chunker)
| std::views::filter(is_token)
| std::views::transform([](auto&& view) { return std::string_view{view}; })
;
}
```
[Which works as expected](https://compiler-explorer.com/z/cPrxTzGf7).
I made two other changes:
1. Instead of taking `input` as a `std::string const&`, I took it as a `std::string_view`… which you should *always* do, unless you *specifically* need a `std::string const&`… which is almost never.
2. Instead of returning a `std::vector<std::string>`… which is *extremely* expensive!… I just return a range of views. This is *far* more efficient, and lazily evaluated. If you *really* want a vector of strings, you can just add `| std::ranges::to<std::vector<std::string>>()` to the end of the chain. (Or, probably better, save the view in a temporary, then use `std::ranges::size()` on it to get the number of tokens, then create a vector with that much space reserved, then `std::ranges::transform()` the view into the vector.)
I also made it `constexpr`, because, why not?
This is not the most efficient solution, especially if the tokenizing function is expensive, because it gets called roughly twice for every character in the input, *plus* roughly once for each token to weed out the whitespace tokens. If the tokenizing function is really cheap, like `is_whitespace()`, or if what you are doing with the tokens is non-trivial and will dwarf the tokenization costs, then that’s probably fine.
## Fixing the original code
If you don’t want to use views, we can fix the original code.
First, we will change the function parameter to be a `std::string_view`.
Next, we’ll replace all the `std::find()` calls with `std::ranges::find_if()`, so we can use a function to determine what is whitespace, rather than a single space character.
The input might have leading whitespace. So we want to skip over that before doing anything else.
Then, as long we’re not at the end, we copy the current token to the output, then skip over any whitespace after it. And we keep doing that.
```
constexpr auto is_whitespace(char c) noexcept
{
return c == ' ' or c == '\t';
}
constexpr auto Foo(std::string_view input)
{
auto output = std::vector<std::string_view>{};
auto const end = std::ranges::end(input);
// Skip leading whitespace.
auto current = std::ranges::find_if_not(std::ranges::begin(input), end, is_whitespace);
while (current != end)
{
// Find the end of the current token, and save it.
auto const current_end = std::ranges::find_if(std::ranges::next(current), end, is_whitespace);
output.emplace_back(current, current_end);
// Skip over any whitespace.
current = std::ranges::find_if_not(current_end, end, is_whitespace);
}
return output;
}
```
[And now that works](https://compiler-explorer.com/z/TjGaKr1K5).
It is theoretically possible to support tokenizing right out of input ranges using something very much like the function above. Rather than the `find_if` function, you would have to use a “copy-while” operation, because you only get one pass over input ranges. And of course, you would have to be copying into a string, not just using views. But that would allow tokenizing directly out of file.
## A generator solution
I’m still not pleased with the original solution constructing a whole vector piecemeal and returning it. I mean, if we *really* want a vector, okay I guess… but if not, it’s such a waste. And if we returned a view or something else lightweight, we can easily construct a vector from it.
Another option is a generator. Very little code needs to change for that:
```
auto Foo(std::string input) -> std::generator<std::string_view>
{
auto const is_whitespace = [](auto c) { return c == ' ' or c == '\t'; };
auto const end = std::ranges::end(input);
// Skip leading whitespace.
auto current = std::ranges::find_if_not(std::ranges::begin(input), end, is_whitespace);
while (current != end)
{
// Find the end of the current token, and save it.
auto const current_end = std::ranges::find_if(std::ranges::next(current), end, is_whitespace);
co_yield {current, current_end};
// Skip over any whitespace.
current = std::ranges::find_if_not(current_end, end, is_whitespace);
}
}
```
The changes I had to make are:
1. Removed the `constexpr`, because coroutines can’t be `constexpr` (yet!).
2. Changed the parameter from a `std::string_view` to a by-value `std::string`. This is not *necessary*, but it is safer. Coroutines can outlive their arguments, so using a view is dodgy. It’s safer to take a copy.
3. Removed the output vector, and everything that changed it or returned it.
4. Added a `co_yield` of the token.
That’s it. [And it works](https://compiler-explorer.com/z/fGvWa66Gq). And while I haven’t profiled it, I would suspect that, especially for input with a lot of tokens, this will probably be faster than returning a vector in most cases, especially if HALO optimization is done… but I can’t say for sure because I think GCC’s support for HALO is still dodgy (and while I think Clang is better, I can’t be arsed to test it right now).
## Maximum efficiency requires a custom view
Everything so far works, and (except for the version returning a vector of strings), it’s all fairly efficient. (And even the vector-of-strings version is efficient if you want a vector of strings in any case.)
But… not yet perfect.
Every solution presented so far has at least *some* inefficiencies:
1. The `chunk_by` solution requires calling the tokenizing function multiple times per character. This is unavoidable when using generic components to build this kind of tokenizer. The desired tokenizer does two things: it splits/chunks the input, and it filters out the whitespace. That’s two jobs, which would require two generic components, and two checks.
2. The generator version requires the coroutine machinery, which *may* be difficult to optimize away. Even if not, it requires copying the input for safety. (If you can risk *not* copying the input, then it could probably be quite efficient.)
3. The (fixed) original code requires building vectors and strings, and doing so eagerly.
> Regarding the solutions in [the existing answer](https://topanswers.xyz/cplusplus?q=749#a874):
>
> 1. The `std::strtok` solution should be less efficient than any of the solutions above. The closest competitor should be the (fixed) original code, but the `std::strtok` solution requires copying the original input (unless it’s already in a `std::string`, it’s mutable and you’re okay with destroying it, in which case you *could* move it rather than copying it).
> 2. The `std::views::split` solution doesn’t work, but if it did (if the results were okay with your expectations), then it *could* be more efficient than anything so far… except for the potentially-unnecessary vector and string constructions.
> 3. The stream iterator solution will be inefficient, although 1) if you really need to read out of an input range (like tokenizing directly out of a file), it’s about as good as you can hope for, except 2) you should either use the original stream, or a [`std::ispanstream`](https://cppreference.com/cpp/io/basic_ispanstream) over string data rather than constructing a `std::istringstream`.
> 4. The regex solution is a non-starter. 1) It is not correct; and 2) even if it were, `std::regex` is a mistake that you should avoid. If you *must* use regular expressions—which is rare in C++—you should find a decent third-party regex library.
For maximum efficiency, you would need a custom view.
I would recommend starting by defining a “tokenizer function” as a function that takes a (forward) range, and returns a sub-view of that range which is the first token.
After that, you need to make a range adaptor—a view—that holds a (view of a) range and a tokenizer function. The view’s iterators could hold a reference to the source view, and a (possibly empty) sub-view that is the “current” token. Each time you increment the iterator, you just call the tokenizer function on a sub-view of the source that starts right after the current token, and replace the current token with the result.
Making a custom view is not trivial, but it’s not *hard*. It mostly involves a *ton* of boilerplate, especially if you want to hook it in to the rest of `std::ranges` functionality (like making it [a RACO](https://en.cppreference.com/cpp/named_req/RangeAdaptorClosureObject)), and *especially* if you want make it maximally powerful (for example, by making it bidirectional when possible; random-access is probably impossible though). But when you’re done, assuming you implemented it efficiently, it would almost certainly be the most efficient solution possible.
With the right tokenizer function, it would possible to use such a view to not only do simple tokenization—like just splitting by whitespace—but even more complex tokenizing; just about any kind of context-free tokenization could be possible. You could even tokenize C++ itself.