Not if “reproduce the [possibly Unicode] string verbatim” includes concatenating it with others in order to insert it into some sort of context, that will absolutely blow up if you’re not careful and unfortunate input data comes in (I hope that happens before the code enters production!).
For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about).
Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding.
Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with:
- strings containing control characters
- strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway)
- strings containing combining characters
- strings in different natural languages
- strings in languages with ligatures
- strings with ligature-like emoji
(and this is obviously not remotely close to exhaustive.)
For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about).
Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)