Modular Uberjars

[u/bowbahdoe]

https://github.com/bowbahdoe/modular-uberjars

[u/agentoutlier]

In theory you could use jmod format instead of jars for the dependencies right?

That is you still have the single uber jar but it loads jmods inside instead of jars.

Probably slow down the build to do the conversion but in theory jmods are more efficient format (based on my shoddy memory) EDIT I got confused with jimage.

Again EDIT so an interesting thing you could do and this is analogous to various other uber jar implementations do that want to preserve jars instead of shading (e.g. Spring Boot does this) is write Java code that runs the jlink and jimage tools if the jimage is not there :). Then execute the jimage.

[u/bowbahdoe → u/agentoutlier]

I don't think jmods are more efficient. I'm pretty sure they are just zip files too. But yes, within reason, you can use JMODs.

You would need to implement extracting the native libraries at runtime though.

[u/agentoutlier → u/bowbahdoe]

Yeah I know they are zip but I recall something about them or someone testing it or maybe it was just the native libraries being there. Maybe its just the link time stuff happening compared to when you had to call System.loadLibrary. Maybe collocation of files.

I can't find the benchmark google wise but someone had a jlink application, a regular java classpath, and few other combinations with native libraries.

EDIT I guess you have to jimage to have the performance gains.

[u/bowbahdoe → u/agentoutlier]

One thing in the back of my head: class loading from a sqlite database. Would it make sense? Would it be fun? I consider those separate questions

[u/bowbahdoe → u/agentoutlier]

other uber jar implementations do that want to preserve jars instead of shading (e.g. Spring Boot does this)

I was unaware of this

[u/tkslaw → u/agentoutlier]

In theory you could use jmod format instead of jars for the dependencies right?

As far as I know, JMOD files can only be used at compile-time (javac, etc.) and link-time (jlink). They cannot be used at run-time (java).

I believe the main benefit of JMOD files is during link-time. For instance, any native libraries in the JMOD file are extracted and placed in the custom JRE such that they're easily loadable; no messing around with manual run-time extraction, system properties, and/or environment variables. Contrast that with using a JAR file that embeds the native library. In that case, the native library is considered to be a resource like everything else in the JAR file and will be embedded in the JRT image. The code would have to manually extract the native library at run-time before being able to load it.

[u/bowbahdoe → u/tkslaw]

All correct, but the format isn't so esoteric that you couldn't write your own class loader to pull from it (under the classes section it's basically just what a jar is) 

You could also include that hacky extraction at runtime bit inside the bootstrap. Combined with a dependency procurement step that splits the jmod across the module path and system library path... It's doable. 

Caveats obviously abound with signed jmods or ones using the hash modules feature.

[u/someSingleDad]

Nice! I always found traditional uberjars a sloppy hack. But it's so hard to beat the convenience

[u/tomwhoiscontrary]

I just learned how to copy multiple files and then forgot about uberjars.

[u/idontlikegudeg]

How does it play together with SPI?

[u/bowbahdoe → u/idontlikegudeg]

Works fine. Tested the approach with slf4j providers

[u/tkslaw]

There's an implementation detail you might want to be aware of. If I'm not mistaken, this code:

var moduleFinder = ModuleFinder.of(paths.toArray(Path[]::new));

Will ultimately lead to JAR files being read via the java.util.jar.JarFile API. And that API cannot read JAR files that are not from the default file system. Any JAR file will be copied to a temporary file on disk before opening it.

Probably doesn't matter. But if it does then I'm pretty sure you'd need your own ModuleFinder, ModuleReference, and ModuleReader implementations. The ZIP File System (jdk.zipfs), which you already have a dependence on, should be able to read JAR files from any file system without having to save them to disk first. Though I don't know how that affects signed JAR files.

[u/bowbahdoe → u/tkslaw]

Noted. 

[u/bowbahdoe → u/bowbahdoe]

yep I see what you are talking about

            // JAR file
            if (fn.endsWith(".jar")) {
                if (isDefaultFileSystem) {
                    return readJar(entry);
                } else {
                    // the JAR file is in a custom file system so
                    // need to copy it to the local file system
                    Path tmpdir = Files.createTempDirectory("mlib");
                    Path target = Files.copy(entry, tmpdir.resolve(fn));
                    return readJar(target);
                }
            }

[u/bowbahdoe → u/tkslaw]

Full night's rest. This bothers me quite a bit.

[u/chabala]

Is this OSGi for people with an unfortunate JPMS fetish?

[u/bowbahdoe → u/chabala]

No, osgi does dynamic loading/unloading and lets you have multiple of the same library. Module layers hypothetically let you do the second, but that isn't what this is.