That's not a UTF-8 issue. That's Unicode. And macOS is not incorrect here.
> So let's say you were looking up something based on the string, it was not there.
What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail.
And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type fi (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi.
> The resolution for now has been using String.Prototype.normalize() with some tricks.
Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "fi" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.
Heh... This reminds me of a bug we ran into a few years ago when rehydrating JSON content for a website my team was working on.
To render a page, we pulled up some JSON data from our database and rendered HTML based on that. To rehydrate the page, we baked the JSON data itself into the page too. We did that by naively JSON.stringify()'ing our database values into a script tag at the bottom of the page. The browser was essentially eval-ing the content in into a variable for us.
But oh how young and naive we were. Before long one of our clients started complaining that our web app was broken for them. We took a look and sure enough, the page was broken. We were scratching our heads for a few hours over that one. The client had somehow managed to insert a weird, invisible character into a string in one of the JSON objects we were rendering. It turns out, contrary to all expectations and sense, JSON isn't actually a strict subset of javascript. There are some perfectly valid JSON values which aren't valid javascript. Somehow our client had accidentally created one such value. When it was rendered back out into our script tag, the browser threw a parse error because our JSON wasn't valid javascript. And that made it abort loading our page's javascript bundle, and everything went sideways.
Of course, the other problem with this approach is that if any string happened to contain "</script>" then the browser would have considered that point the end of the javascript content. That would have also done weird, wild and potentially dangerous things to our page. But luckily we found that before any of our users stumbled on it. At least, as far as we know.
I've noticed on several occasions that text written by Mac users and published online would replace the Swedish letters ä and ö with a¨ and o¨. I think it's due to some combination of how they do Unicode normalization and how Firefox on Windows renders the text. I haven't seen it for at least a year, so I'm guessing it's resolved now.
Same with Firefox on MacOS. Chrome and Safari do find "fi" for "fi" and "fi" for "fi".
This suggests some questions.
1. Are fi.com and fi.com the same as far as DNS in concerned, or could both exist as separate websites?
2. If they are separate websites, do the various password manager browser extensions recognize they are different, or might they fill in your fi.com password when you are getting phished with fi.com? Same question for the various browser's built-in password and form savers.
Thanks for trying to be helpful. Of course, I over-simplified our 4 years of development and the specialized field we are in. There isn't really a database involved here. I am sure there are many more correct ways for us to do certain things, that said, I respectfully disagree and macOS is the issue here.
When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.
> When you can be compatible with everything else, please try to be. MacOS isn't in this case.
You say that like macOS is bucking the trend. NFC is a very common normalization form for text input, regardless of whether the browser chooses to normalize. And older versions of the W3C "Character Model for the World Wide Web: String Matching" note[1] even recommended using NFC explicitly, although the current version says that skipping Unicode normalization is the recommended form of matching for "new specifications".
As for Safari, it's been doing NFC normalization ever since 2006[2], and it sounds like this was originally at least partially-motivated by fixing a compatibility issue with Windows.
In any case, Safari converting text to NFC is certainly differing behavior from non-WebKit browsers, but if you give someone text in NFD and they delete and retype it, it's likely to end up in NFC anyway, which means this is a problem regardless of browser.
> So let's say you were looking up something based on the string, it was not there.
What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail.
And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type fi (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi.
> This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...
That article flips NFC and NFD.
> The resolution for now has been using String.Prototype.normalize() with some tricks.
Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "fi" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.