Update Git author script

During the svn => git sync process we rewrite the author using a bash script that fetches the user details. This is get-author.sh in the git home directory on svn servers.

Due to character sets, it appears that non-latin names are being corrupted.
For example:

$ git clone git://develop.git.wordpress.org/
$ cd develop.git.wordpress.org
$ git show f047b94d71e780cbd7595047f28a644955d35fff | head -n3
commit f047b94d71e780cbd7595047f28a644955d35fff
Author: Greg ZiÃ³Å‚kowski <gziolo@git.wordpress.org>
Date:   Fri Apr 21 10:41:58 2023 +0000

Greg ZiÃ³Å‚kowski should be Greg Ziółkowski as shown on his profile.

The SQL used for this is CONCAT(display_name, '|', user_nicename). I can’t test it as I don’t have mysqlMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. tools on my sandbox, but I suspect either

Character sets need to be specified on the mysql command, I suspect either --default-character-set=latin1 or --default-character-set=utf8mb4 would work.
The above concat should do some character-set conversions; I think CONCAT( CONVERT( CAST( CONVERT( display_name USING latin1) AS BINARY) USING utf8), ‘|', user_nicename) would work.

To duplicate it, you should be able to run this on the svn host:
get-author.sh gziolo

Note: You can likely remove the 2015-era logging/debugging from the file at the same time, I don’t recall the outcomes of that, but I suspect it was long fixed.. Review the logs I guess!

Let me know if you’d like me to test or debug anything.

cc @dmsnell @gziolo (Apologies for the months long delay!)
#prio2 #git #svn

Dennis Snell 10:06 am on March 21, 2024
To share a tiny bit of context, in Greg’s case, it seems like his name is properly stored in UTF-8 in the source, as we can work backwards from the corruption to see what happened.

I’ve made a diagram to try and show what’s happening.

https://fluffyandflakey.blog/wp-content/uploads/2024/03/utf8-double-encoding.png

Something must be casting text that’s already UTF-8 into UTF-8, which means it tries to encode the UTF-8 bytes directly as if they were code points.

This runs into trouble with the final 0x82 byte because that’s in the set of C1 control characters. It took me a while to figure this out, because it’s not valid latin1. I think that one of several things is happening:
- if the input character encoding is listed as cp1252 (windows 1252) then most of the C1 control characters are remapped to higher Unicode code points.
- if the conversion library is treating the input as HTMLHTML HTML is an acronym for Hyper Text Markup Language. It is a markup language that is used in the development of web pages and websites., then even if the input character encoding is listed as UTF-8, if a numeric character reference encodes the C1 control character range, those will be remapped as is the page were in cp1252. if the library treats raw bytes in the same way as it would numeric character references then this would lead to the seen transform.
- I’ve seen converters apply this extra logic when working with other character encodings, to my dismay (here’s looking at Node’s TextDecoder which doesn’t know the difference between ASCII, latin1, and cp-1252). if the converter conflates them, we’ll be in this situation.
It’s the final , which gives this away as it should be mapping to U+142 for the ł. Every other transform has a correspondence with latin1, but not this one.

As a prediction, if someone were to have the letter É in their name, I would expect the process to corrupt it into Ã‰, since É‘s UTF-8 bytes are C3 89, and if treating them as cp1252 code points would be U+C3 U+2030 (through the C1 remapping).
Dennis Snell 8:36 am on April 10, 2024

While discussing this problem yesterday with Alex Kirk, he had an idea which led to a much clearer resolution.

Before calling SELECT CONCAT(… we run SET CHARSET 'latin1';. This effectively tells MySQLMySQL MySQL is a relational database management system. A database is a structured collection of data where content, configuration and other options are stored. https://www.mysql.com/. to stop re-encoding the text for the consuming connection, preserving the UTF-8 bytes stored as latin1 text.

While the CAST() method you propose @dd32 looks right, adding a quick selector before the query is a bit easier on the eyes. I think both solutions are valid. Maybe semantically the casts are more self-documenting, but the SET CHARSET is less distracting.
- Dion Hulse 1:05 am on April 11, 2024
  
  I’m about 98% sure that we don’t need to do any CAST or SET CHARSET and setting the character set on the CLICLI Command Line Interface. Terminal (Bash) in Mac, Command Prompt in Windows, or WP-CLI for WordPress. will properly handle it (mysql --default-character-set=latin1)
Dennis Snell 7:29 pm on August 12, 2024

While this doesn’t resolve the problem where it starts, we can work around the bug and tidy up past mistakes by adding a .mailmap file on the git side. This not only fixes names corrupted by the author script, but also merges commits from the same person who committed with different identities over time.

Site resources