[Native Image] JDK23+ jdk.internal.org.jline* does not work for Windows

[Karm]

Describe the Issue

Hello,

As I have been debugging build time issues:

• Windows native: JLINE(JANSI), ForeignAPISupport, JDK 23+ quarkusio/quarkus#43777
• originally reported as GraalVM 23 on Windows: VMError$HostedError: should not reach here: unexpected input could not be handled: linkToNative (possibly caused by Quarkus setup) #9727

I noticed this runtime issue where terminal escape sequences are ignored in a terminal that cannot handle those and would require Kernel32 API native API calls to e.g. set colors.

Consider this Java code:

public class Main {
    public static void main(String[] args) {
        final char[] redLineBytes = new char[] {
                0x1B, 0x5B, 0x33, 0x31, 0x6D, // ESC[31m Set to red
                0x52, 0x45, 0x44, // "RED"
                0x1B, 0x5B, 0x30, 0x6D  // ESC[0m Reset
        };
        try (final java.io.PrintWriter w = System.console().writer()) {
            w.write(redLineBytes);
            w.println(System.lineSeparator() + "NOT RED");
            w.flush();
        }
    }
}

CMDER https://cmder.app/

Cmder terminal understands ESC sequences, so it interprets them and it works nicely both in HotSpot and native:

C:\tmp
λ native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)

C:\tmp
λ javac Main.java

C:\tmp
λ java -agentlib:native-image-agent=config-output-dir=AGENT Main
RED
NOT RED

C:\tmp
λ native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=./AGENT -H:+ForeignAPISupport Main
Warning: The option '-H:+ForeignAPISupport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'main' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                   (12.1s @ 0.06GB)
 Java version: 23+37, vendor version: Mandrel-24.1.0.0-Final
 Graal compiler: optimization level: 2, target machine: native
 C compiler: cl.exe (microsoft, x64, 19.36.32535)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 1 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 8.81GB of memory (55.1% of 16.00GB system memory, determined at start)
 - 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis...  [*****]                                                                   (10.8s @ 0.31GB)
    3,283 reachable types   (71.4% of    4,599 total)
    3,852 reachable fields  (43.5% of    8,848 total)
   15,510 reachable methods (44.7% of   34,727 total)
    1,021 types,    13 fields, and   140 methods registered for reflection
       61 types,    51 fields, and    52 methods registered for JNI access
        0 downcalls and 0 upcalls registered for foreign access
        1 native library: version
[3/8] Building universe...                                                                               (1.8s @ 0.36GB)
[4/8] Parsing methods...      [*]                                                                        (1.3s @ 0.40GB)
[5/8] Inlining methods...     [***]                                                                      (1.1s @ 0.44GB)
[6/8] Compiling methods...    [***]                                                                     (10.1s @ 0.38GB)
[7/8] Laying out methods...   [*]                                                                        (1.7s @ 0.42GB)
[8/8] Creating image...       [**]                                                                       (2.4s @ 0.50GB)
   5.45MB (40.77%) for code area:     8,885 compilation units
   7.75MB (57.96%) for image heap:  106,145 objects and 1,979 resources
 173.48kB ( 1.27%) for other data
  13.37MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   4.04MB java.base                                            1.43MB byte[] for java.lang.String
   1.01MB svm.jar (Native Image)                               1.40MB byte[] for code metadata
 114.06kB java.logging                                       999.47kB java.lang.String
  70.33kB org.graalvm.nativeimage.base                       782.64kB java.lang.Class
  50.67kB jdk.proxy2                                         304.25kB byte[] for general heap data
  40.44kB jdk.proxy1                                         282.13kB com.oracle.svm.core.hub.DynamicHubCompanion
  26.96kB jdk.internal.vm.ci                                 271.08kB java.util.HashMap$Node
  22.05kB org.graalvm.collections                            262.34kB java.lang.Object[]
  12.43kB jdk.proxy3                                         183.54kB java.lang.String[]
   8.13kB jdk.graal.compiler                                 164.83kB byte[] for embedded resources
   3.28kB for 5 more packages                                  1.75MB for 935 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
                        2.1s (4.5% of total time) in 364 GCs | Peak RSS: 0.92GB | CPU load: 5.14
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
 C:\tmp\main.exe (executable)
========================================================================================================================
Finished generating 'main' in 43.9s.

C:\tmp
λ main
RED
NOT RED

CMD

Good old cmd terminal you find on all Windows does not understand ESC sequences and it requires new JDK internal JLine (formerly standalone lib) to interpret ESC codes (e.g. WindowsAnsiWriter.java) into Kernel32 API calls with which one can control the terminal.

The bug is that while HotSpot does this, native-image does not go this FFM code path, does not call to Windows Kernel32 API and just dumps the bytes to the terminal:

C:\tmp>native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)

C:\tmp>javac Main.java

C:\tmp> java -agentlib:native-image-agent=config-output-dir=AGENT Main
RED
NOT RED

C:\tmp> native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=./AGENT -H:+ForeignAPISupport Main
Warning: The option '-H:+ForeignAPISupport' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'main' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                   (13.1s @ 0.06GB)
 Java version: 23+37, vendor version: Mandrel-24.1.0.0-Final
 Graal compiler: optimization level: 2, target machine: native
 C compiler: cl.exe (microsoft, x64, 19.36.32535)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 1 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 8.81GB of memory (55.1% of 16.00GB system memory, determined at start)
 - 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis...  [*****]                                                                   (12.3s @ 0.32GB)
    3,283 reachable types   (71.4% of    4,599 total)
    3,852 reachable fields  (43.5% of    8,848 total)
   15,510 reachable methods (44.7% of   34,726 total)
    1,021 types,    13 fields, and   140 methods registered for reflection
       61 types,    51 fields, and    52 methods registered for JNI access
        0 downcalls and 0 upcalls registered for foreign access
        1 native library: version
[3/8] Building universe...                                                                               (1.9s @ 0.37GB)
[4/8] Parsing methods...      [*]                                                                        (1.4s @ 0.39GB)
[5/8] Inlining methods...     [***]                                                                      (1.1s @ 0.43GB)
[6/8] Compiling methods...    [***]                                                                     (10.4s @ 0.35GB)
[7/8] Laying out methods...   [*]                                                                        (1.6s @ 0.43GB)
[8/8] Creating image...       [**]                                                                       (2.3s @ 0.50GB)
   5.45MB (40.77%) for code area:     8,885 compilation units
   7.75MB (57.96%) for image heap:  106,150 objects and 1,980 resources
 173.50kB ( 1.27%) for other data
  13.37MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   4.04MB java.base                                            1.43MB byte[] for java.lang.String
   1.01MB svm.jar (Native Image)                               1.40MB byte[] for code metadata
 114.06kB java.logging                                       999.50kB java.lang.String
  70.33kB org.graalvm.nativeimage.base                       782.64kB java.lang.Class
  50.67kB jdk.proxy2                                         304.25kB byte[] for general heap data
  40.44kB jdk.proxy1                                         282.13kB com.oracle.svm.core.hub.DynamicHubCompanion
  26.96kB jdk.internal.vm.ci                                 271.08kB java.util.HashMap$Node
  22.05kB org.graalvm.collections                            262.34kB java.lang.Object[]
  12.43kB jdk.proxy3                                         183.54kB java.lang.String[]
   8.13kB jdk.graal.compiler                                 165.82kB byte[] for embedded resources
   3.28kB for 5 more packages                                  1.74MB for 935 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
                        2.3s (4.7% of total time) in 367 GCs | Peak RSS: 0.88GB | CPU load: 5.20
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
 C:\tmp\main.exe (executable)
========================================================================================================================
Finished generating 'main' in 46.8s.

C:\tmp>java Main
RED
NOT RED

C:\tmp>main
�[31mRED�[0m
NOT RED

JDK21 Optional

For JDK 21 HotSpot, you can switch the jline capability on like this:

C:\tmp>       java -Djdk.console=jdk.internal.le Main

Agent

The difference between what the native agent generates as reachability json is quite subtle:

CMDER

{
  "resources": [
    {
      "glob": "META-INF/services/java.lang.System$LoggerFinder"
    },
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jansi"
    },
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jna"
    },
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jni"
    },
    {
      "module": "jdk.internal.le",
      "glob": "jdk/internal/org/jline/utils/capabilities.txt"
    }
  ],
  
  "bundles": []
}

CMD

{
  "resources": [
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jansi"
    },
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jna"
    },
    {
      "glob": "META-INF/services/org/jline/terminal/provider/jni"
    },
    {
      "module": "jdk.internal.le",
      "glob": "jdk/internal/org/jline/utils/capabilities.txt"
    },
    {
      "module": "jdk.internal.le",
      "glob": "jdk/internal/org/jline/utils/windows.caps"
    }
  ],
  
  "bundles": []
}

Reproduce it

SET JAVA_HOME="C:\tmp\graalvm-community-openjdk-24+17.1\"
SET GRAALVM_HOME=%JAVA_HOME%
SET PATH=%JAVA_HOME%\bin;%PATH%
vcvars64
native-image --version
javac Main.java
java -agentlib:native-image-agent=config-output-dir=AGENT Main
native-image --no-fallback --link-at-build-time -march=native -H:ConfigurationFileDirectories=AGENT -H:+ForeignAPISupport Main

CI

You need an actual terminal. If you merely run it in a CI headless, there is no terminal to talk to.

I will keep digging...

WIP

Using the latest version of GraalVM can resolve many issues.

• I tried with the latest version of GraalVM.

GraalVM Version

C:\tmp>native-image --version
native-image 23 2024-09-17
OpenJDK Runtime Environment Mandrel-24.1.0.0-Final (build 23+37)
OpenJDK 64-Bit Server VM Mandrel-24.1.0.0-Final (build 23+37, mixed mode)

and

C:\tmp>native-image --version
native-image 24 2025-03-18
GraalVM Runtime Environment GraalVM CE 24-dev+17.1 (build 24+17-jvmci-b01)
Substrate VM GraalVM CE 24-dev+17.1 (build 24+17, serial gc)

Operating System and Version

Windows 2019 Server

[Karm]

I'll try to adjust it with RuntimeForeignAccess.registerForDowncall