在 Java JNI 中获取真正的 UTF-8 字符

Question

有没有一种简单的方法可以在 JNI 代码中将 Java 字符串转换为真正的 UTF-8 字节数组？

不幸的是 GetStringUTFChars() almost 完成了所需但不完全是，它 returns 一个 "modified" UTF-8 字节序列。主要区别在于修改后的 UTF-8 不包含任何空字符（因此您可以将其视为 ANSI C 空终止字符串），但另一个区别似乎是如何处理 Unicode 增补字符，例如表情符号。

一个字符如 U+1F604 "SMILING FACE WITH OPEN MOUTH AND SMILING EYES" 被存储为一个代理对（两个 UTF-16 字符 U+D83D U+DE04）并且有一个 4 字节的 UTF-8 相当于 F0 9F 98 84，这就是我在 Java:

中将字符串转换为 UTF-8 时得到的字节序列

    char[] c = Character.toChars(0x1F604);
    String s = new String(c);
    System.out.println(s);
    for (int i=0; i<c.length; ++i)
        System.out.println("c["+i+"] = 0x"+Integer.toHexString(c[i]));
    byte[] b = s.getBytes("UTF-8");
    for (int i=0; i<b.length; ++i)
        System.out.println("b["+i+"] = 0x"+Integer.toHexString(b[i] & 0xFF));

上面的代码打印以下内容：

c[0] = 0xd83d c[1] = 0xde04 b[0] = 0xf0 b[1] = 0x9f b[2] = 0x98 b[3] = 0x84

但是，如果我将 's' 传递给本机 JNI 方法并调用 GetStringUTFChars()，我将获得 6 个字节。每个代理项对字符都被独立地转换为 3 字节序列：

JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s)
{
    const char* sBytes = env->GetStringUTFChars(_s, NULL);
    for (int i=0; sBytes[i]!=0; ++i)
        fprintf(stderr, "%d: %02x\n", i, sBytes[i]);
    env->ReleaseStringUTFChars(_s, sBytes);
    return result;
}

0: ed 1: a0 2: bd 3: ed 4: b8 5: 84

Wikipedia UTF-8 article 表明 GetStringUTFChars() 实际上是 returns CESU-8 而不是 UTF-8。这反过来导致我的本机 Mac 代码崩溃，因为它不是有效的 UTF-8 序列：

CFStringRef str = CFStringCreateWithCString(NULL, path, kCFStringEncodingUTF8);
CFURLRef url = CFURLCreateWithFileSystemPath(NULL, str, kCFURLPOSIXPathStyle, false);

我想我可以更改我所有的 JNI 方法以采用 byte[] 而不是 String 并在 Java 中进行 UTF-8 转换，但这看起来有点难看，有更好的解决方案吗？

Answer 1

这在 Java 文档中有清楚的解释：

JNI Functions

GetStringUTFChars
const char * GetStringUTFChars(JNIEnv *env, jstring string, jboolean *isCopy);
Returns a pointer to an array of bytes representing the string in modified UTF-8 encoding. This array is valid until it is released by ReleaseStringUTFChars().

Modified UTF-8

The JNI uses modified UTF-8 strings to represent various string types. Modified UTF-8 strings are the same as those used by the Java VM. Modified UTF-8 strings are encoded so that character sequences that contain only non-null ASCII characters can be represented using only one byte per character, but all Unicode characters can be represented.

All characters in the range \u0001 to \u007F are represented by a single byte, as follows:

The seven bits of data in the byte give the value of the character represented.

The null character ('\u0000') and characters in the range '\u0080' to '\u07FF' are represented by a pair of bytes x and y:

The bytes represent the character with the value ((x & 0x1f) << 6) + (y & 0x3f).

Characters in the range '\u0800' to '\uFFFF' are represented by 3 bytes x, y, and z:

The character with the value ((x & 0xf) << 12) + ((y & 0x3f) << 6) + (z & 0x3f) is represented by the bytes.

Characters with code points above U+FFFF (so-called supplementary characters) are represented by separately encoding the two surrogate code units of their UTF-16 representation. Each of the surrogate code units is represented by three bytes. This means, supplementary characters are represented by six bytes, u, v, w, x, y, and z:

The character with the value 0x10000+((v&0x0f)<<16)+((w&0x3f)<<10)+(y&0x0f)<<6)+(z&0x3f) is represented by the six bytes.

The bytes of multibyte characters are stored in the class file in big-endian (high byte first) order.

There are two differences between this format and the standard UTF-8 format. First, the null character (char)0 is encoded using the two-byte format rather than the one-byte format. This means that modified UTF-8 strings never have embedded nulls. Second, only the one-byte, two-byte, and three-byte formats of standard UTF-8 are used. The Java VM does not recognize the four-byte format of standard UTF-8; it uses its own two-times-three-byte format instead.

For more information regarding the standard UTF-8 format, see section 3.9 Unicode Encoding Forms of The Unicode Standard, Version 4.0.

由于U+1F604是增补字符，而Java不支持UTF-8的4字节编码格式，所以U+1F604通过UTF-16代理对编码在修改后的UTF-8中表示U+D83D U+DE04 每个代理使用 3 个字节，因此总共 6 个字节。

所以，回答你的问题...

Is there an easy way to convert a Java string to a true UTF-8 byte array in JNI code?

您可以：

使用 GetStringChars() 获取原始的 UTF-16 编码字符，然后从中创建您自己的 UTF-8 字节数组。从 UTF-16 到 UTF-8 的转换是一种非常简单的手动实现算法，或者您可以使用您的平台或第 3 方库提供的任何预先存在的实现。

让您的 JNI 代码回调到 Java 以调用 String.getBytes(String charsetName) 方法将 jstring 对象编码为 UTF-8 字节数组，例如：

JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s)
{
    const jclass stringClass = env->GetObjectClass(_s);
    const jmethodID getBytes = env->GetMethodID(stringClass, "getBytes", "(Ljava/lang/String;)[B");

    const jstring charsetName = env->NewStringUTF("UTF-8");
    const jbyteArray stringJbytes = (jbyteArray) env->CallObjectMethod(_s, getBytes, charsetName);
    env->DeleteLocalRef(charsetName);

    const jsize length = env->GetArrayLength(stringJbytes);
    const jbyte* pBytes = env->GetByteArrayElements(stringJbytes, NULL); 

    for (int i = 0; i < length; ++i)
        fprintf(stderr, "%d: %02x\n", i, pBytes[i]);

    env->ReleaseByteArrayElements(stringJbytes, pBytes, JNI_ABORT); 
    env->DeleteLocalRef(stringJbytes);
}

The Wikipedia UTF-8 article suggests that GetStringUTFChars() actually returns CESU-8 rather than UTF-8

Java 的修改后的 UTF-8 与 CESU-8:

不完全相同

CESU-8 is similar to Java's Modified UTF-8 but does not have the special encoding of the NUL character (U+0000).

在 Java JNI 中获取真正的 UTF-8 字符

Getting true UTF-8 characters in Java JNI

java

java-native-interface

encoding

utf-8