Updated! (June 26, 2022) The article has been updated to use JEP 424 build 19-ea+25 (2022-09-20).

Introduction

In this series of articles, we will explore the APIs from OpenJDK’s Project Panama. My intent is to show you how to be proficient in using the Foreign Function and Memory Access APIs (‘java.lang.foreign’) as it relates to Java interoperability with native code.

While this article is for newbies, I assume you know the basics of the Java language, a little bash scripting, and a familiarity with C programming concepts. If you are new to C language don’t worry I will go over the concepts later.

For the impatient go to Panama4Newbies on GitHub.

Note: To see the prior code examples of this article using JEP 419 go to the branch here.

In Part 1 of this series, I will give you an overview of the requirements and later go through some exercises as a primer to call C functions.

What is Project Panama?

Project Panama is a new way for the Java programming language to access native libraries that are written in native languages like C, C++, and Fortran (currently supports C). While my description might be an oversimplification, this project has been in the making for quite some time (4-8 years) and continues to have great success! Kudos and a huge shout out to the OpenJDK community at large!

Project Panama is an umbrella project that comprises many JEPs (Java Enhancement Proposal) as shown below:

• Foreign Function and Memory Access APIs
  • 1st Incubator JEP-412
  • 2nd Incubator JEP-419
  • Preview JEP-424
• Foreign-Memory Access API - is a way to allocate and map memory outside of the JVM’s heap. Capable of creating C pointers, structs, and other primitive or native data types: JEP-370, JEP-383
• Foreign Linker API - is a way to access functions in native libraries: JEP-389
• Vector API - a platform agnostic way to utilize vector hardware such as SIMD (Single Instruction Multiple Data) chips: JEP-338

Shown below is the timeline of the roadmap:

As you can see on the timeline for Feb. 2022 is JEP 424. This enhancement proposal addresses its preview release where package namespaces get rolled up into Java proper java.lang.foreign.

Note: This tutorial series is based on JEP 424

Why Use Project Panama?

Before answering why, let me ask another question, Have you heard of Minecraft? If you answered "Yes, of course", then, great! If you answered "No", then please go and visit https://www.minecraft.net/en-us. Did you also know that Minecraft is written in Java using the popular open source gaming library called LWJGL (LightWeight Java Game Library)?

So, where am I going with this? As you dig deeper into the gaming library’s implementation you’ll discover that it uses JNI (Java Native Interface) to access native libraries such as OpenGL, OpenCL, etc. These native libraries are written in C (language) capable of accessing hardware on a device such as sound, input devices, or GPUs.

So, back to the answer to "Why?". The short answer is: Panama is simpler to use and can have better performance.

The longer answer is that using JNI's generated code and wrapper code can be error prone, and is often difficult to maintain over time. Also, it requires natively compiled glue (wrapper) code to be installed on the system (which means you’ll need system administrative privileges). In other words, project Panama is a pure Java solution that allows you to access existing native libraries with comparable or better performance than JNI.

Here are the use cases for using Project Panama:

• Access C code from Java
• Access device drivers on embedded environments such as Raspberry Pi
• Access memory off the JVM’s heap
• Increased performance using lower level APIs to access SIMD hardware.
• Up calls or Callbacks from C code to Java code

Okay, so now that you are convinced, where can you get Project Panama?

Where do I get Project Panama? (jextract Tool)

Go to https://jdk.java.net/panama/ to download the Early-Access JDK builds for your operating system. As the time of this writing an early-access build of JDK 19-ea containing jextract tool has not been built. You will have to download JDK 19-ea, and see the separate instructions to build jextract by yourself below.

NEW! The jextract tool is released as a GitHub project separately. By being independent from the OpenJDK project it will allow the tool's release to line up with a particular JDK build release starting with JDK 18. Check out my article on how to build jextract yourself.

Why not just get the official GA release?

Important: While the official GA releases of OpenJDK 19+ contains the Panama APIs, they will not contain the jextract tool that is going to be used in this tutorial.

As we explore further, we will learn that the jextract tool is responsible for generating Java (code) bindings derived from C header files and their associated native library files. Library files with the extension .dll, .so and .dylib are used on the Windows, Linux, and MacOS operating systems respectively.

Getting Started

Let us make sure the environment is setup before we begin. The following are install instructions for your respective OS.

Mac OS X / Linux setup instructions:

Step 1: Download JDK 19-ea from https://azul.com/downloads or https://jdk.java.net/19/ and untar or unzip into a directory.

Step 2: Follow instructions to build jextract here . After you create the tool based on your JDK downloaded from Step 1, the runtime image will reside in jextract/build/jextract directory.

Step 2: Set JAVA_HOME and PATH

 # Mac
 $ export JAVA_HOME=<parent directory>/jextract/build/jextract
 
 # Linux
 $ export JAVA_HOME=<parent directory>/jextract/build/jextract
  
 $ export PATH=$JAVA_HOME/bin:$PATH

Note: To make environment variables permanent you can set these in your .bashrc, .bash_profile files on Linux or MacOS respectively.

Step 3: Test runtime and jextract is available

 $ java -version
 $ jextract -h

Windows instructions:

Step 1: Download and untar or unzip into a directory.
Step 2: Set JAVA_HOME and PATH

 c:\> set JAVA_HOME=<unzipped_dir>\jdk-19
 c:\> set PATH=%JAVA_HOME%\bin;%PATH%

Note: To make environment variables permanent on the Windows platform do the following:

1. Right-click the Computer icon and choose Properties, or in Windows Control Panel, choose System.
2. Choose Advanced system settings.
3. Relaunch a command prompt (cmd.exe)

Step 3: Test runtime and jextract is available

 c:\> java -version
 c:\> jextract -h

After running jextract -h to display the switch options you’ll know you are ready to go. You should see something like the following:

Usage: jextract <options> <header file>                                  

Option                         Description                               
------                         -----------                               
-?, -h, --help                 print help                                
-D <macro>                     define a C preprocessor macro             
-I <path>                      specify include files path                
--dump-includes <file>         dump included symbols into specified file 
--header-class-name <name>     name of the header class                  
--include-function <name>      name of function to include               
--include-macro <name>         name of constant macro to include         
--include-struct <name>        name of struct definition to include      
--include-typedef <name>       name of type definition to include        
--include-union <name>         name of union definition to include       
--include-var <name>           name of global variable to include        
-l <library>                   specify a library name or absolute library path   
--output <path>                specify the directory to place generated files    
--source                       generate java sources                     
-t, --target-package <package> target package for specified header files 
--version                      print version information and exit 

If you are seeing the jextract options then you are ready to go to the next section.

Do I need a C compiler?

In short, No. For demonstration purposes I will be using a standard C compiler on my MacOS environment. This is purely optional and I will use it to show concepts inside a C program.

If you are on a Windows OS you can check out Microsoft’s Visual C++ that includes a their C compiler (https://docs.microsoft.com/en-us/cpp/build/walkthrough-compile-a-c-program-on-the-command-line?view=msvc-160). Also, you can download MingW at https://www.mingw-w64.org

Let’s Do It!

Before we get into using Panama’s APIs let's begin by looking at a Hello World example in the C programming language. By understanding what a C program consists of this will help us know how to invoke native code. Later on we will write a pure Hello World Java program that will call into standard C (native) functions.

Anatomy of a Hello World in C

Note: Remember, this part of the tutorial is purely optional if you don’t have a C compiler handy for your OS just skip the compilation step.

Step 1: Enter the listing 1 into an editor and save the file as helloworld.c.

Listing 1: helloworld.c

#include <stdio.h>
int main() {
   printf("Hello, World! \n");
   return 0;
}

Step 2: Compile the code

$ gcc helloworld.c

Step 3: List the executable file

$ ls -l a.out
-rwxr-xr-x  1 jdoe  staff  49424 Jul 29 21:06 a.out

Step 4: Run or execute the program

$ ./a.out
Hello, World!

Well that was pretty straight forward! Let’s unpack what is actually going on. Below is a high-level look at what the C program is doing.

• Include statement using Ansi C’s stdio.h library header. Similar to Java’s imports.
• The main() function is the entry point similar to Java’s public static void main() method
• The main() function has a C int return type.
• The body calls the stdio’s printf() function that takes a const char * type.

Here are more details on the four observations:

In Step 1 the stdio is C’s standard input output library. In the C programming language files with .h are similar to Java’s interfaces where it defines or describes function signatures and constants of a library. The stdio library in the example contains the printf() function equivalent to System.out.printf() in a Java program.

In Step 2 the C programming language it has two overloaded functions of main(). One takes an empty parameter signiture, and the other will take number of args (type int) and an array of type char * (C string).

int main() {}
int main(int argc, char *argv[]) {}

In Step 3 the main() function will return an integer of zero to denote success, and any other value is an error status code.

In step 4 The stdio.h header function int printf(const char *format, ...), has a signature that takes a C string type with variable arguments (0-to-many) values to perform a string interpolation. Similar to Java’s System.out.printf(“Hello, %s\n”, “Hello Panama!”);.

There are two things to keep in mind when talking to C.

• Be aware of C #includes. Header files on your system will allow jextract to generate code bindings (class files).
• Know that C’s datatypes will need to be converted between Java and C as needed. Luckly, Panama will create convience methods that makes this easy!

Now that we have a good understanding of a Hello World C program let’s create an equivalent Panama Java Hello World example. In other words the Java program will natively call the printf() function.

Panama Hello World Example

As mentioned before, the C code of our Hello World program was using the stdio.h library. At this point, we will generate Java code to talk to the stdio.h library. Instead of hand coding this (which is out of the scope of this tutorial) we will be using the jextract tool. This magical tool will generate pure Java code that will bind to native libraries. These class files will contain meta data and much of the lower level Panama code that will make things convenient for the user of the API (you and me).

Note: When using jextract's switch --source it can emit or generate source code that can be used in projects. Here's where you can view Panama specific code.

Because the C language specification is well defined jextract can generate Java code pretty easily. C++ on the other hand will be another effort and is not currently supported. If you have a native library written in C++ you’d have to take some additional steps which I won’t cover.

Let’s jextract STDIO please!

To use jextract let’s locate where your stdio.h file is located.

Step 1: Find header file stdio.h

$ gcc -H -fsyntax-only helloworld.c

On MacOS (Big Sur, Monterey) the output looks something to the following:

. /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h
.. /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h
...

Now that you know where the file is located, we can target the file when using the jextract tool.

$ jextract [options] <path_to_file/stdio.h>

Step 2: Use jextract to generate Java code from stdio.h header file.

On MacOS do the following:

$ jextract --source --output src -t org.unix -I /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h

On Linux:

$ jextract --source --output src -t org.unix -I /usr/include /usr/include/stdio.h

It should look like the following:

$ ls -l src/org/unix

Now we can use the generated code to be used in our Panama Java Hello World program.

Step 3: Create a Java HelloWorld.java file

Copy and paste the following into a file HelloWorld.java.

import static java.lang.foreign.MemorySegment;
import static java.lang.foreign.MemorySession;
import static org.unix.stdio_h.printf;

public class HelloWorld {
    public static void main(String[] args) {
       try (var memorySession = MemorySession.openConfined()) {
           MemorySegment cString = memorySession.allocateUtf8String("Hello World! Panama style\n");
           printf(cString);
       }
    }
}

Step 4: Running the Panama Java HelloWorld.java

$ java --enable-native-access=ALL-UNNAMED --enable-preview --source 19 HelloWorld.java

The output is the following:

Hello World! Panama style

How does it work?

Step 1, the C compiler on MacOS/Linux platforms allows you to find where headers are located on a system. Often MacOS developers will use Brew (package manager) to install 3rd party libraries such as OpenCL, Tensorflow, etc.

Step 2 This is where the magic happens. The jextract tool will generate the source code using --source. The -t is the output directory or namespace on the class path. These classes can be jarred up to be distributed for your app. -I (dash capitol ‘i’) specifies the directory of include files. If you want to target other directories just specify additional -I <dir_path_to_header_files> (Include switches).

The targeting file path to the header file (stdio.h) is the last argument to jextract.

What about 3rd party C libraries?

When 3rd party libraries are installed they typically are in your library path such as /usr/lib or /usr/local/lib. To specify a library the -l (lowercase 'L') option is used. The value will be the name of the library or the absolute path to the library. For example, say you want to use Tensorflow (Google's Machine Learning library).

On the Mac OS the file would be named libtensorflow.dylib and on Linux should be tensorflow.so. I believe on Windows OS it should be named tensorflow.dll.

To specify -l option (L) you can specify the name of the library or the absolute path of the library file. For example, to jextract Tensorflow it will look like the following:

$ jextract \
  -I /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ \
  -t org.tensorflow \
  -I ${LIBTENSORFLOW_HOME}/include \
  -l ${LIBTENSORFLOW_HOME}/lib/libtensorflow.dylib \
  ${LIBTENSORFLOW_HOME}/include/tensorflow/c/c_api.h

You will notice on MacOS I specified the fully qualified library file ${LIBTENSORFLOW_HOME}/lib/libtensorflow.dylib instead of the name (tensorflow). Usually, if libraries are installed in /usr/lib or /usr/local/lib you can just specify the name.

Step 3: Java code talks to C.

public class HelloWorld {
    public static void main(String[] args) {
       try (var memorySession = MemorySession.openConfined()) {   // (A)
           MemorySegment cString = memorySession.allocateUtf8String("Hello World! Panama style\n"); // (B)
           printf(str);  // (C)                                                       
       }
    }
}

In line (A) the statement is where the code uses a try memory session to create a scope. A scope will auto close when its finished the try-block. This will deallocate native memory safely.

The statement in line (B) calls the allocateUtf8String() method to allocate and convert a Java String into a C string (char * type). Any time you are mimicking C variables they will be of type MemorySegment. In part 2 of this series we will look at C pointers, so if you find the asterisk beside the char * type a little odd, don't be too concerned about it for now, and just know that it is the C language's way of storing a string value (or array of characters).

In statement (C) the call to the native function printf() with the object cString of type MemorySegement is passed in and invoked. Another important note to understand is stdout (Standard output) in C, will need to be flushed if you have a Java System.out.println() For instance:

var cString = memorySession.allocateUtf8String("Hello World\n");
printf(cString);
System.out.println("Java System.out\n");

One some systems, (on my MacOS) the following output occurs because of the C's output hasn't been flushed:

Java System.out
Hello World

So, what do you do? You need to call fflush()to flush C's buffer to stdout.

var cString = memorySession.allocateUtf8String("Hello World\n");
printf(cString);
fflush(NULL());
System.out.println("Java System.out\n");

Now, the output will be the following:

Hello World
Java System.out

So, now that you know how to create a C string to call a native C function, let's see how to use a C string (char *) to print using Java's System.out.printf().

public class HelloWorld {
    public static void main(String[] args) {
       try (var memorySession =  MemorySession.openConfined()) {
           // C's string to Java String using Java's printf
           MemorySegment cString2 = memorySession.allocateUtf8String("Panama");
           String string2 = cString2.getUtf8String(0);
           System.out.printf("Hello, %s from a C string. \n", string2);
       }
    }
}

The output is:

Hello, Panama from a C string.

Let's look at more examples of creating different C primitive types.

Creating C primitive data types

To make this easy to remember think of Allocator -> MemorySegment -> get/set or Allocate -> MemSeg pattern. When creating C primitive variables remember that they are like objects where space is allocated and has a memory address to access the value (dereference).

The following example is creating a C double and assigning it a Java double. (Code uses JAVA_DOUBLE when the jextract generated C_DOUBLE variable wasn't created).

var cDouble = memorySession.allocate(JAVA_DOUBLE, Math.PI);
var msgStr = memorySession.allocateUtf8String("A slice of %f \n");
printf(msgStr, cDouble.get(JAVA_DOUBLE, 0));

On the last line above, you'll notice the cDouble.get(JAVA_DOUBLE, 0) instead of C_DOUBLE. This is because the C_DOUBLE isn't available until jextract generates it for you. e.g. C_DOUBLE$LAYOUT is a JAVA_DOUBLE.withBitAlignment(64).

Note: It's better to use the jextract generated C_DOUBLE because it takes on the underlying OS' bit width. It might not occupy the same number of bits if you are assuming it's 64 bits. The following is the C_DOUBLE variable of type OfDouble that gets generated by jextract.

// jextract generated C_DOUBLE
public static OfDouble C_DOUBLE = Constants$root.C_DOUBLE$LAYOUT;

// In Constants$root from generated code
static final  OfDouble C_DOUBLE$LAYOUT = JAVA_DOUBLE.withBitAlignment(64);

To be clear, it's better to generate code that defines a C_DOUBLE instead of using the above JAVA_DOUBLE. The following code is preferred:

var cDouble = memorySession.allocate(C_DOUBLE, Math.PI);
var msgStr = memorySession.allocateUtf8String("A slice of %f \n");
printf(msgStr, cDouble.get(C_DOUBLE, 0));

The output is the following:

A slice of 3.141593

Above code listing first creates an SegmentAllocator, next it allocates (create space) a MemorySegment to hold a C double type using the SegmentAllocator's allocate() method. The first parameter takes a MemoryLayout, in this case there are all C data types, that are statically defined in the Linker class. The second parameter takes a Java primitive value, in this case I used Math.PI to be assigned.

Creating C arrays

Now that you know how to create primitive data types let's create C arrays. Shown below is allocating space to hold a single dimesional array off of the Java heap. Then we just re-access the C array and display the contents.

System.out.println("An array of data");
MemorySegment cDoubleArray = memorySession.allocateArray(C_DOUBLE, new double[] {
       1.0, 2.0, 3.0, 4.0,
       1.0, 1.0, 1.0, 1.0,
       3.0, 4.0, 5.0, 6.0,
       5.0, 6.0, 7.0, 8.0
});

for (long i = 0; i < (4*4); i++) {
   if (i>0 && i % 4 == 0) {
       System.out.println();
   }
   System.out.printf(" %f ", cDoubleArray.get(C_DOUBLE, i * 8));
}

The output:

An array of data
 1.000000  2.000000  3.000000  4.000000 
 1.000000  1.000000  1.000000  1.000000 
 3.000000  4.000000  5.000000  6.000000 
 5.000000  6.000000  7.000000  8.000000 

As seen above the use of MemorySegment.get() method can grab the value as a Java primitive to be displayed.

Note: New to JDK 18+ (JEP 419) are get and set methods introduced into the MemorySegment class.

Conclusion

In Part 1, we learned about the what, where, and whys regarding project Panama. Next, we examined the anotomy of a typical Hello World C program. After learning how to use jextract to generate Java code from stdio.h, we were able to create a Java Hello World to access the C function printf(). Lastly, we learned how to create C primitive data types including arrays.

If you're still interested, in Part 2 we will continue our journey into Project Panama, by looking at C's concept of structs and pointers.

Resources

• Panama4Newbies: https://github.com/carldea/panama4newbies
• OpenJDK's Project Panama - https://openjdk.java.net/projects/panama/
• Project Panama EA Builds - http://jdk.java.net/panama/
• Mailing list: - https://mail.openjdk.java.net/mailman/listinfo/panama-dev
• Fosdem 2022 Project Panama talk
  Native Language Access: Project Panama for Newbies (Carl Dea): https://youtu.be/REyl4cOsItE
• Panama Examples: https://hg.openjdk.java.net/panama/dev/raw-file/4810a7de75cb/doc/panama_foreign.html#using-panama-foreign-jdk
• Minecraft - https://www.minecraft.net/en-us
• LWJGL and JNI - https://github.com/LWJGL/lwjgl3/blob/master/config/macos/build.xml
• JNI: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/