OpenJDKPodcastSecurity

Foojay Podcast #104: 53 CVEs, Two Emergency Releases: Why Java Security Holds Up with Simon Ritter

Foojay Podcast #104: 53 CVEs, Two Emergency Releases: Why Java Security Holds Up with Simon Ritter

53 CVEs in one quarterly update, followed by two emergency releases within days. That is a lot of issues in a short time, but it is also a sign that the system works: AI tools are now analyzing Java, the problems get found and fixed first, and a whole community of engineers and commercial distributors is behind every patch.

In this episode, Simon Ritter, Deputy CTO at Azul, Java Champion and member of the OpenJDK Vulnerability Group, explains what happened over the past weeks. He is normally invited twice a year for a new Java release, but this time the security news could not wait.

We start with what a CVE is and how a vulnerability travels from a report to a patch, in a closed and encrypted group, before anyone can exploit it. From there we cover why the number of fixes jumped from an average of 7.6 per quarter to 53, how the release cycle is moving toward monthly security updates, the eMudhra certificate problem that caused two out-of-cycle releases, and the backporting work behind every supported Java version. We close with free versus paid OpenJDK builds, the CPU, PSU and CSPU abbreviations, and the one thing to remember: check your Java version and be ready to update at any moment.

If you want to understand how CVEs are handled behind the scenes and what the different types of security updates actually mean, this episode explains it.

YouTube

Podcast Apps

You can listen and subscribe to the Foojay Podcast on:

Guests

Content

  • 00:00 Introduction of topic and guest
  • 01:59 What is a CVE and how are they reported and handled?
  • 08:45 How the Java security release cycle is changing and each have more CVEs
  • 11:48 How AI impacts the discovery of CVEs
  • 14:09 What was the eMudhra problem causing two out-of-cycle security updates?
  • 21:49 The impact of creating an out-of-cycle update (and backports)
  • 26:05 How safe will OpenJDK builds be in the future? And dependencies?
  • 29:06 Free versus paid versions of OpenJDK builds
  • 32:40 CPU vs PSU vs CSPU: the names used for different types of security updates (security only versus including patches)
  • 35:52 What will happen with old Java versions like 6, 7, 8
  • 37:05 Why would you pay for a supported OpenJDK build?
  • 41:50 Conclusion: check your Java version and be ready to update at any moment!

Transcript

Automatically generated from the audio, so it can contain errors in names and technical terms. Suggest a correction.

[0:00] 53 CVEs followed by two emergency releases. What happened with Java in the past 2 weeks?

[0:07] Welcome to the Foojay podcast. All your news about OpenJDK.

[0:14] Welcome to another episode of the Foojay podcast. Let’s talk about security. We are recording this on Friday, October the 30th, and you may notice, if you watch this podcast on YouTube, that I have some more gray hair compared to the previous episode. The last 2 weeks I have been busy with new release notes one after the other for various libraries, projects, and runtimes. Most of these related to issues found by AI, yes, of course, and being exploited within hours. So, keeping your runtime, code, and dependencies up to date has become even more critical than ever. If you want to know more about the security issues and updates related to Java, check my Foojay blog of Friday about the Azul Zulu updates and the post by my guest of today published on last Wednesday about why unsupported OpenJDK has become a security gamble. Welcome to the podcast, Simon. You are deputy CTO at Azul, a Java champion, and one of the people who has followed the Java runtime the longest in our community.

[1:21] So, normally I invite you to the podcast twice a year when we have a new Java version. but with all that has happened in the last few weeks, I wanted to invite you again on short notice. So, thank you for joining to explain us what was going on in the past weeks. So, let’s start with a small review. We had the scheduled October update, which was released on the 20th. And it fixed record record number of CVEs, 53. On average, we have about seven or eight. so, let’s start with the basic. What is a CVE and how does it involve a Java release or an update?

[2:04] Right. So, a CVE stands for common vulnerability and exposure. And it’s essentially a way that we track problems related to security in software. And where somebody identifies something that could be a vulnerability in terms of it could be exploited to allow remote code execution or denial of service, data breach, and so on. So, CVE is essentially a description of what the problem is. But, if you look on the CVE website, you’ll actually find that there’s not a lot of detail.

[2:41] Mhm.

[2:41] And that’s quite deliberate because of course we don’t want to say the problem with this particular vulnerability is because XML parsing allows you to inject code and you know, if you give a very detailed description, people can then more easily find exploits for that. So, what you will find if you look at the CVEs is it will just say something along the lines of there is a vulnerability that affects the Java runtime. It won’t say where in the Java runtime it is. But, the real detail associated with that CVE, which is not made public, is what gets used to then identify how to resolve that. And another important thing about CVEs is they have a score. So, there is a CVSS, common vulnerability scoring system.

[3:25] And that is a number between well, 0.1 because 0 would mean there is no vulnerability. So, a number between 0.1 and 10.0 and that gives you an idea of how bad that vulnerability is. And they grade them as low, medium, high, and then critical. So, the critical ones are the 9.0 to 10.0. Those are the ones which are the ones people most worry about because of course where you have a critical vulnerability, it could be something that’s very easy to exploit. You know, we use example of Log4Shell, Log4j. That was a 10.0, absolute critical vulnerability because it was very easy to exploit. Needed to be patched very quickly. High ones obviously are not quite so bad, but they’re still if if you’re looking at whether you should patch things, high ones definitely need to you need to look at what you’re doing with those.

[4:18] Medium ones, yeah, sure, you can maybe take more interest in that. And low ones, you might think, “Oh, well, low ones, we don’t need to worry about that.” But, what we’re seeing now is AI.

[4:30] Mhm.

[4:30] And the problem is that where you whereas before you might think, “Well, mediums and lows, they don’t have a lot of problem because, you know, they don’t allow you to do anything particularly dangerous.” But what AI is now doing is that chaining together some of these smaller ones to create a vulnerability from, let’s say, a five, a three, and a four and then you create a vulnerability which does allow you to create a denial-of-service attack or a remote code execution. And that is very worrying.

[4:59] And the whole thing with So, the CVE number is where everything starts. So, someone finds an issue, reports it. I create release notes for Java builds. At the time that I create release notes, some of these I cannot access them myself. If I go to the website, it just says, “This number was reserved.” So, it doesn’t even give you a title yet. So, how do you if you’re involved in this fix or fixing this issue, how do you know what’s inside the ticket?

[5:30] Right. So, this is where we see and if we talk about Java specifically, within the OpenJDK project, there is a group called the OpenJDK Vulnerability Group. And this is a closed group, even though it’s an open-source project, it’s a closed group, invitation only, you have to be somebody who understands security, you have to be approved by the members of this. Oracle started this a good few years ago when they moved to the six-month release cadence, they changed the licensing rules of the Oracle JDK, and with all credit to Oracle, they realized that there would be a proliferation of OpenJDK distributions. And one of the things that they were very concerned about was making sure that we could have the maximum level of security for Java.

[6:16] And that would involve sharing information with people who are creating these distributions. So, as all obviously, people like Amazon because they have Corretto, people like Microsoft because they have their own JDK, SAP, Red Hat, and so on. And so, they created this closed group and they invited people from these different organizations who had responsibility for security. And then, the vulnerability that you see as a reserved number, we really don’t want to provide any information about that at all because, of course, we’re in the process of developing a patch for it and it hasn’t yet been released. If somebody could see details, they could potentially develop an exploit and then that becomes a zero-day vulnerability. So, we’re very, very careful about not making that information public before a patch is released.

[7:06] What we would do is within the OpenJDK vulnerability group, the information about that CVE is available. Somebody reports a CVE. We get the information about that, which particular part of the JDK is affected, and then we will look at what we can do to resolve that, create a patch which fixes the flaw, and then that will be released as one of these patches that we get in the scheduled updates.

[7:33] And I can confirm that the system works because I’m really the last point before documentation goes out. Me and my colleague [clears throat], we really have to publish this is the list of CVEs that was fixed in this release. We only get this list at the moment we need to push the button. Now you can publish the post. It’s It’s a really secure and closed community that is fixing the things, eh?

[7:56] It is. And even to the point where all of the email between the members of this group because I one of the members of this group all of the email between the members of this group is encrypted, because we don’t want to have a situation where somebody might be able to intercept email about, you know, the vulnerability group and therefore be able to figure out what’s going on. So, we keep all of the information secure. None of the email that’s shared between the group is published. So, again, even though it’s an open project, we don’t publish the mailing list like we do with other things. So, it’s very, very tightly controlled with the idea that we only want people to know about these vulnerabilities when we have a patch available. And that means that you have the opportunity to patch it immediately before people have a time to start, you know, trying to exploit it.

[8:45] Okay. then we have the three months so until, let’s say, half this year, we had a three-month release cycle.

[8:53] Yes.

[8:54] Every three months, a new update of existing versions was published with all the fixes. What happened that we now suddenly end up with 53 CVEs in October?

[9:05] Right. So, as you say, we have up until now, we have had quarterly updates, which were in January, April, July, and October. And very nicely scheduled the third Tuesday of the month, you knew that like clockwork there would be an update that would come out and that would contain a number of security patches, bug fixes, maybe some other performance improvements and things like that. So, it was very predictable. What we’ve seen is that if you go back before July, which is when we started to see the real impact, I would say, of AI being applied to a lot of different software packages and specifically Java. [snorts] If we go back before July and go back several years, you’d find that the average number of CVEs that have been addressed in each update is 7.6.

[9:54] So, just under eight CVEs. Now, that’s kind of a fair number because, yes, Java is 30 years old. It’s 7 1/2 million lines of code. We keep adding new features to it. So, it’s impossible to say we can eliminate all the vulnerabilities, but eight per quarter is a nice kind of number you’re not too worried. And in fact, if you go back, we’ve not had any critical vulnerabilities for quite a lot number of years. We’ve had a few highs, but we haven’t had any critical vulnerabilities in the JDK itself for a number of years. What has now changed is that if you look at the July release, there were I think it was 16 Yeah, I think it was 16 in July. I have to double check that. but essentially, July suddenly we saw twice as many as the average in that one release.

[10:49] And then because Oracle is aware that this is going to increase, so we saw 53 in the October release. We do we’re seeing a very steep increase. They’ve also made the decision that we’re going to move away from a quarterly update to monthly updates. Right now, we’re not doing every month, but [clears throat] probably as we move into the next calendar year 2027, we will see that we will get to a point where we’re seeing updates every month for security patches. We won’t see bug fixes every month. That will still be on the quarterly update, but we will see specific updates released that address security vulnerabilities. We’ve already seen that in August, so that was the first out of quarter [snorts] update, which was a monthly one. They’re calling these CSPUs, critical security patch updates.

[11:41] Just security patches in between the scheduled quarterly updates.

[11:47] Okay. so, are people using AI to find these? That That’s a bit the idea. So, people are really investigating the tools they are using so that they are now finding issues is a good thing. So, the whole thing gets [clears throat] better. It’s just the amount of issues which is found that is a problem.

[12:08] Yeah, I mean this is what we’re seeing is that because earlier this year the real kind of tipping point or changing point if you like was the idea of Anthropic real announcing they didn’t release it. They announced Claude Mythos. It’s a large language model frontier model which was tuned specifically around the idea of software and identifying security vulnerabilities and being able to exploit them. So they created this as a model. They started looking at open source software. They turned it on all the common operating systems, all the common browsers as a starting point and they found vulnerabilities in every single one of them. If you look at the sort of history of this, they say that the oldest vulnerability they found was 27 years old and that was in OpenBSD.

[12:59] This means there’s a vulnerability that had been in OpenBSD for the nearly 30 years and nobody found it until they turned Mythos onto it and it identified this thing. So this has led to the increase that we’re seeing now. Now this is technically a good thing because what’s happening is it’s not malicious actors who are using this to identify vulnerabilities. The reason we’re seeing a significant increase right now in terms of the updates to Java is that Java is being analyzed by these large language models in order to identify where these vulnerabilities are. So this means that we can then patch those and make those patches available for people using other large language models have an a chance to try and identify these vulnerabilities and then exploit them.

[13:49] So it is technically a good thing. It just means that we’re facing I’ve heard it described as a tsunami of vulnerabilities and patches.

[13:58] Mhm. And yeah, release notes at the Saturday night at 3:00 in the morning. I can testify about that. so yeah, and that 3:00 in the night I think was then the Immuta certificate issue which is not really a Java problem. It’s a certificate What are these certificates used for? It’s It’s to chain messages and say, “Yeah, this is really coming from this person.”

[14:27] Yes. So, if we hold that for just a moment and talk a little bit about the fact that this is a very significant thing as well in that when we talk about vulnerabilities that are in Java, you might think, “Oh, okay. Well, it’s vulnerabilities in the Java code so it’s the JVM, the libraries, and so on.” Which is absolutely correct. There are often vulnerabilities that we find there. But, you have to remember that OpenJDK is a full software package, the platform that you run applications on. And so, it isn’t just about looking at those 7 and 1/2 million lines of code that we include in OpenJDK. You also have to consider that there are lots of libraries which are used to build things like the JVM for the Windows platform or for Linux or for Mac. Those libraries may well contain vulnerabilities. And if those contain vulnerabilities, we then need to update the library that we’re using to build OpenJDK to eliminate that. So, there’s there’s that’s a knock-on effect plus what we’ve seen with the issue issue around certificates, which is something which is not part technically of the JDK itself.

[15:32] We include these certificate authorities as part of the ability to run secure transmission over network. So, there’s there’s SSL and TLS, secure socket layer, transport layer security. Very commonly used when it comes to web servers and you have the obvious like HTTP and HTTPS. HTTPS is the one we always use or should be using, which is the idea of secure transmission. And that relies on these TLS and SSL underlying libraries. But, in order to do that securely, you also need the trust between the two endpoints of a communication. And that’s where these certificate authorities come in. I don’t go into all the details of all how this actually works, but essentially what you get in the JDK is a set of certificate authorities which are trusted trusted providers of these certificates could then be used to set up a secure communication between two endpoints and then you have encryption and so on that works with that.

[16:40] Now, what happened with eMudhra was that there were I think it was eight of these root certificates as they’re called were compromised. again, I I’m not a security expert from the level of security of certificates and so on, but I was reading that apparently this Now, I might be wrong on this, so if if I am wrong on this, you know, just forgive me. But I believe that in order to create these certificate authorities certificates, there’s like a 384-bit number that you can use to generate random numbers. A lot of random numbers involved in this. The certificate authority only used the first byte, so they had they literally used 1 to 255, which meant that it was very easy to compromise those certificates because people just use [clears throat] the numbers 1 to 255 rather than trying to guess a 384-bit number.

[17:33] And that made meant they could then get access to the information about these certificate authorities and so on. That meant that we had to do something very quickly because, it exposed a problem where that secure communication that you would have between two endpoints for a network connection could potentially be compromised by what’s called a man-in-the-middle attack. If you have access to the information about those certificates, you could basically intercept information coming from your web browser to pick up the information from the web browser, then pass it on to the server, but you’re then able to decrypt the information as it passes from the browser to the server. Man-in-the-middle attack, if you’ve got the certificate authority information, then that becomes easy to do. So, that’s very, very serious and that was a 10.0 as a vulnerability. So, really had to get that and it was a zero-day vulnerability, meaning that it was exposed as a vulnerability before patches had been made available.

[18:38] Is that a bad decision by someone? I found this, I just publish it or

[18:44] There was again, I what I read there was an issue where people who were responsible for addressing this, especially through things like browsers, because the browser is clearly very important because that one end point of a lot of communication. So, there was a plan to roll this out, but one particular company, whom I won’t name, released information about it. I think it was 6 hours before the general information was supposed to go live. And that did cause some problems in terms of people being able to get out information before the web browser providers had been able to address this issue through pushing out the information to browsers.

[19:27] Yeah. So, so we’re talking about this issue related to out-of-cycle Java updates. It but it wasn’t actually a Java issue. It was a much broader problem, yeah.

[19:37] Yeah. And so, this is what happened is and this is where it becomes very, very critical to have access to these updates is because as you said, it was an out-of-cycle one. We literally Tuesday, we released 53 patches to vulnerabilities that had been identified. And then I think it was Thursday when this all kind of came about and it was like suddenly it’s like, “Ooh, we need to patch this very, very quickly.” And there was one patch that was created which was the idea of invalidating these certificates through the digital signature that they used for it. [snorts] Which obviously solved the problem. But then somebody else looked at it and went, “Ooh, actually, even if you invalidate certificates, if you regenerate the certificate using a different random number I can’t, again, I can’t remember all the exact details, but there was a thing where Sonatype came up with this thing they said if you regenerate those keys with a different number, even though you’ve invalidated this fingerprint for the old ones, those new ones are now valid again and it will get missed by the fix. So we then had to

[20:46] Come up with another fix which is where you ended up on was it Friday night, Saturday night? yeah, we had to come up with another fix which basically had to remove the certificate authorities from the it’s called the CA certs file in the JDK. And that clearly meant that even if you change the numbers associated with the certificates, they were no longer valid. So that was the thing that had to be done. So it was a second again it was a 10.0 I think CV that they came out with literally in 2 days.

[21:16] Yeah, the timing is a bit let’s call it funny. So as I said, I’m only at the end of the of the chain. I just publish release notes like for the Azul builds. I think at the moment we said yeah, let’s publish the October release. I think the first message appeared about the Imudra thing. And then we when we published the fix a few days later again at the moment we said that let’s publish this someone said yeah, but I think I found another way to exploit this. So it’s yeah, it’s fun to see how this happens. On the other hand it’s frightening that it goes so fast. But I’m also happy to see how fast the whole chain worked to really report this that it ends up with the right people and that’s like at Azul and I guess also at the other Java distributors a whole battery of systems is started to build new versions, test them on different systems. It’s a massive work behind the scenes, eh?

[22:24] It is and I would say it’s it’s one of those things that we are going to have to see how this works in practice as we move forward because already with 53 vulnerabilities in one update, this is placing a significant additional workload. Although there’s those 53 vulnerabilities that have been identified and patched. Now, you have to remember that the way this works with OpenJDK is those vulnerabilities are identified and patched in the current release. So, that’s OpenJDK 27. Now, we have these long-term support releases which are the ones that Oracle and other OpenJDK distribution providers provide extended maintenance and support for. So, there’s 25, 21, 17, 11, 8, and in the case of Azul, 7 and 6.

[23:17] So, there are a number of older versions which have extended maintenance and support. Now, when we have seven, eight vulnerabilities in a quarter, it’s not too much work most of the time to say, “Okay, we’ll take those vulnerabilities that have been identified in OpenJDK 27, then we can backport those to 25, 21, 17, and so on.” When we got 53, 53 times however many versions you want to support, that becomes a lot more complicated because even though a lot of these patches they apply very easily, some of them will because it’s just the same code base from let’s say JDK 17 as it is in JDK 27. So, the patch applies very quickly, very easily. There’s no no issue with that. But sometimes, because things change between JDK 17 and JDK 27, when you try to backport the patch, you find that it’s not simply a case of just overlaying the same code, and you’ve then got to work through it and figure out,“Oh, okay. Now, we’ve got something that’s different.” and we need to figure out how to deal with that and then apply the patch. So, it does the patch apply cleanly is a real issue.

[24:32] And as I say, the more patches we have, the more long-term support versions we have as well, because that’s increasing technically. it means the workload on the team who are dealing with these vulnerabilities and backporting them so that distributions can provide updates to 17 or 11 or 8. that is starting to become something that we need to look very carefully at. do we have enough resources

[24:57] Mhm.

[24:58] To deal with that?

[24:59] Yeah. And I guess the people in those teams are really good merge, rebase, cherry-pick experts because this is a mess, I think.

[25:08] It absolutely is. I mean, I say cuz I’m on the on this group and it’s very interesting to see the email that goes back and forth. And I’m not going to say anything about what’s in that email, but there’s some very skilled people. And they they’ve been doing this for a long time. And I do kind of wonder what happens as those people start to retire. [laughter] Again, it’s this, this issue of like: when, when we’re not bringing in Well, not so much when we’re not bringing juniors in at the bottom of the chain how do we develop people with the senior skills to, to then be able to fill the place of that or substitute in to get these things working? So, there’s a lot of things to consider there.

[25:48] Okay. You cannot mention what is in the emails, but how much swearing is there in the in these emails [laughter]?

[25:56] Oh, no.

[25:58] Well, at least there will be if as soon as it turns out,“Yep, we found another one.” Yeah, OpenJDK builds, they have been actually safe for the long time, which is three every three months this update. Is this now broken? Do we need Do devops teams need to be ready to update at any point?

[26:24] This is something that people are going to have to face, end users. And this it not just a Java thing at all. This is across the whole IT landscape is that we are going to have to think more about our patching strategy. With Java in the past, it’s been fairly easy because you go,“Okay, every quarter an update comes out contains a number of security patches.” You could look at the level of those patches, what CVEs, and you could say, “Okay, well, we’ve only got some medium ones in this one. We haven’t even got any high ones. So, let’s not worry too much about deploying this patch or update immediately. We can get to it later on.“But, with now a monthly cadence potentially for updates which could contain high or even critical vulnerabilities, then the users of not just Java but Java libraries and other things are going to have consider very carefully how they approach patching and how quickly they can deploy these patches to get around the problems of potentially AI exploits.

[27:30] Mhm. Yeah, and I’m we’re only looking at it now from the point of view of a the Java runtime. [clears throat] But, also yeah, there are issues in dependencies and in the code itself. So, yeah, we need to rethink about how easy can we deploy a new version on a new runtime with new dependencies and still be sure that it works as expected.

[27:53] Yeah, and there’s there’s even bigger problems than this. I mean, Steve Poole has written a couple of very good articles around some of the open source side of things. And if you think about Java, Java’s a very well supported project. You’ve got lots of big companies involved, obviously Oracle, Red Hat, IBM, SAP, Amazon, Microsoft. There’s all, you know, lots of people involved in that. Lots of engineering effort going into that. If you look at, you know, open source libraries, there may be a lot of libraries that are no longer maintained.

[28:26] No.

[28:26] And if you’re using those libraries and nobody’s maintaining it, or you’ve got It’s It’s the diagram, isn’t it, which everybody loves where you’ve got all the different pieces and there’s one little piece at the bottom which holds everything up and they say"maintained by one guy in Nebraska.” [laughter] and that’s that’s so true, you see, because if a vulnerability is detected in something which doesn’t have a maintainer [snorts] to it, then what do you do?

[28:54] Yeah. And it’s also the fact that some libraries move to a new version of Java and if your application is stuck at an older version

[29:02] Exactly. Yes.

[29:03] They have this dependency issue, yeah. you also wrote something about in your article, that’s a nice sentence,“free as in puppy, not free as in beer.“It was about the runtime. So, OpenJDK is an open source project, you can get builds for free, but you can also pay a company to provide you the latest and newest build with the fix faster.

[29:25] Mhm.

[29:26] Is this an important topic now?

[29:29] It is and it becomes even more important now because the point I was trying to get across there is if you think about it, a lot of people think, okay, you’ve got OpenJDK is a free open source project. Great. So, everybody can look at the source code, everybody can download the source code, you can build your own OpenJDK distribution. Literally, you know, it’s really easy because you just, you know, do make on it. You do build make and you can run the scripts. But, the issue that people are facing is with free builds, that doesn’t mean that there isn’t some work involved in that. So, there has to be somebody somewhere building your distribution that you’re going to use. And there has to be somebody somewhere who’s potentially running tests on it because if you are using Java, really you should be using a TCK tested JDK distribution in order to guarantee that you’ve got the maximum level of compatibility and you don’t need to worry about chain making any changes.

[30:29] So, realistically, you should be using a TCK tested build of Java as a runtime. So, there’s the build process and there’s the running all the TCK tests and then the distribution and so on. That is not free. Somebody has to do it. It takes resources. You need machines to do the builds, especially if you’re supporting a large number of distribution or large number of versions, large number of platforms. The matrix becomes really big and how much resource you need to do that, not just in terms of machine time, but also the engineers you need to do that. that starts to add up and that is not free. So, if you’re getting a free distribution, somebody somewhere has to pay for all that work. A lot of the time, it’s people like Amazon, it’s people like Azul, it’s people like Microsoft, where you have a commercial distribution, but you can make an unsupported free version available really as a sort of byproduct of the commercial version. But for ones where that’s not the case, it is more of an issue that you need to think about as to how that work actually gets done and how quickly it can get done.

[31:38] Yeah, the numbers are if you just think about so you have Linux, Mac, Windows, you have arm processors, you have Intel processors, then you have Java 6 still. So, that’s the matrix you’re talking about. You end up with hundreds of configurations.

[31:55] Yeah.

[31:55] Then you have a JDK, you have a GRE, so the just a run time. If you multiply that, it’s hundreds of configurations that all need to be built and tested.

[32:06] And you might even for older versions, you might have both 64 and 32-bit as well for Intel. But as you say, yes, the numbers become really quite significant in terms of hundreds, literally hundreds of builds. And if you’re doing that on a quarterly basis, you might be able to, you know, have enough resources to do that. If you’re switching to a monthly basis, that’s then three times as much work that you need to get done.

[32:33] Yeah, and last week we had monthly and then two out of cycle, so that

[32:36] Again, yes.

[32:37] Also adding to it. You have these these CPUs. So we there are a few abbreviations used here and they change a lot in the last months. So you have the CPU critical patch update, you have a PSU or patch set update and you have the

[32:53] CSU.

[32:55] Yes.

[32:56] How are they all related and what should we know about them?

[33:00] Yes. This is where [laughter] This is where things have got complicated because we they we tend to use some of these terms interchangeably and not necessarily in the right way. So the what we used to have was and this is using Oracle terminology for how they provide the updates to their software is you used to have what was called a PSU which was a patch set update. And that was the full update. So it contained the security patches, it contained the bug fixes, any performance improvements, any source code changes. So it was like a big chunk of changes, quarterly update. Then they had thing called the CPU or critical patch update which was only the security patches. The idea behind that was if you’ve got a large number of changes in a full update, the PSU, you can run into problems where changes that you put into that fix certain bugs can then have an impact on stability of other things and we’ve seen that reasonably frequently. About one in five updates, the full update introduces a regression against something somewhere.

[34:06] And they have to potentially change that again to fix the fix. [snorts] So the CPU is about having the smallest possible set of changes that you can then roll out as quickly as possible because you need those security updates. And then the CSPU or critical security patch update is this new monthly update which is only security patches. So it’s like the CPU but it’s the monthly one in between the quarterly ones.

[34:31] Mhm.

[34:32] I’m sure that’s clear as as crystal. [laughter]

[34:36] Yeah, I think the naming is under investigation. Let’s say.

[34:39] Yes. Especially because a lot of the [snorts] distributions refer to the CPU as what they provide even though it’s actually the full update and it’s technically what would be described as the PSU. So, yeah, there’s like I said, these things get used interchangeably and you need to look carefully at what you’re actually getting.

[34:57] Yeah. So, for companies with a big farm of applications running, it’s it’s good to look at can we step into some cycle where we get this critical update thing only

[35:11] Yes.

[35:12] We can test it that we can deploy it very fast, confirm it works correctly, and then we have some time some more time to install the full update and be ready for the next cycle. And yeah, again we used to have 3 months to do that. And now it turns into

[35:30] Technically we still do because remember that the CSPU only contains security patches. So, the full update is only every quarter even though we’re switching to a monthly release for security, the bug fixes and so on are still on a quarterly basis. So, we don’t have quite so much to worry about that even though it still doesn’t make life too easy for us.

[35:50] Okay. You mentioned I think in the beginning you still have builds for Java 6 and 7. Will they still be maintained for a long time as fast as as the other ones?

[36:05] Yes. again, it’s it’s one of those things where you have to start looking at you you’re now on JDK 27, we’re still supporting JDK 6 so that’s 21 versions later. the issue I guess is that, you know, when we’re back porting those those security patches, we only need to back port the ones that are applicable to JDK 6 meaning that if there’s a feature that was introduced like modularity for example. Modularity came in JDK 9 so there is no modularity in JDK 6. If there’s a bug found in modularity, there’s nothing to back port because there is no modularity feature in JDK 6. So, the number of patches that you end up back porting sort of is on a slope. So that you get less as you go further back.

[36:50] Okay. But maybe a shout out to everyone with outdated Java versions. That may be a good use case for AI to bring your [laughter] your application to a newer version. Okay. Now put on your sales hat. Oh, why I let you do some promotion now. why should companies look for a paid distribution? What is the benefit there?

[37:21] Yeah. I mean there’s two very obvious benefits. One is that if you have a paid distribution, you will have a some assuming you’re using Azul, you will have a service level agreement. That service level agreement is a guarantee that once the embargo is lifted on the update, and whether that’s quarterly or a monthly update. the idea of the embargo is that there are lots of distributions and they all have their own update, but nobody can release their update until Oracle has released theirs. And that’s when the embargo is lifted. So Oracle releases their update, everybody else can then follow after that. Now the SLA basically dictates how quickly after that or how much time can pass before you get access to the update.

[38:10] Now in terms of Azul, I think our SLA is 48 hours, meaning that 2 days within 2 days of Oracle’s update coming out, you will get access to the update. We actually exceed that a long way. So to date, we’ve had our updates available within 1 hour of Oracle releasing theirs. And that is significant because it means that there is no window of opportunity for people to exploit these things before you have access to the updates and are able to deploy them in your estate. So that’s one very key thing is how quickly you can get access to updates. In the case of other distributions, you may be waiting a few days, you may even be waiting over a week. you know, there are if it’s free build, nobody’s guaranteeing when or even if you’ll get that update. So, there’s there’s that to consider.

[38:59] The other thing that’s very important is that, going back to this idea of stability, if there’s a change in an update which introduces a regression, but that regression only affects your application. And this is quite conceivable cuz I’ve actually seen it with some of our customers. So, somebody changes something in update in let’s say July 2026, and that affects your application. So, your application doesn’t work the way you expect it to. Now, what do you do? If you’re concerned about security, you need to install the security updates. So, if you’ve got access to the CPU and only the security updates, then you can continue to do that. But if you’ve only got the full update, which changes the way your application works, you’re stuck with you can’t install it because it breaks your application, but you need to install it because it fixes security problems.

[39:51] If you’ve got support, not only will you get those updates quickly, but you’ve also got somebody to call and say,“Hang on, my application doesn’t work with this update. I need somebody to fix it.“And this is what our support organization will do. They’ll say,“Oh, okay. Let’s look at what’s going on with your application, figure out what changed in that update, and then we can issue a separate build just for your application, make sure your application gets working again, and then we’ll upstream any change into OpenJDK so that we don’t have to maintain a fork of OpenJDK [snorts] specifically to our given customer. So, there’s a number of advantages in terms of having paid support.

[40:32] Mhm. And that’s not only for the Java runtime, it’s also for a lot of open source libraries also have this model where you have you can pay the creator of a library if you find an issue that they help you faster. That’s a bit the same idea. You have a direct contact, someone you can call because yeah, I love open source. I have a lot of open source. That’s also the strength of Java OpenJDK that there’s a whole community behind it and pushing this forward. But yeah, all a lot of these people, it’s it’s it’s a side project for them. So, they’re not available whenever you need them and that’s a yeah, again, like you said, this whole diagram of dependencies and then at the bottom you have but like like the log for shell issue, the few people that were involved in the project, yeah, it was weekend long and a week long that they only had to deal with this.

[41:33] They Yeah, even with the given the fact that they have a job and so they had to drop everything to fix this and yeah, it’s the recurring problem in everything which is open source is who are you going to call? Now I sound like Ghostbusters, but [laughter] that was not my idea. Okay, okay, a lot of info. The important message here is yeah, check your Java version. What’s the most important thing that people should remember?

[42:01] I think the most important thing that we need to realize from the last few months is you really need to be on top of your patching. You really need to be making sure that your systems are updated and keep an eye on what’s happening with those updates when they come out as to what vulnerabilities. Not so much from the detail of what those vulnerabilities are cuz as I say, you don’t get much information about that, but just the number, the score that’s associated with that because the higher the score, the more dangerous they are. But even so, with AI now chaining together things, you’ve got to remember that high score doesn’t actually or a low score doesn’t necessarily mean that it it’s something that can’t be exploited.

[42:42] Okay, and also review your whole build pipeline and release pipeline and how do you deploy things and are you able to go to the best and latest and the fixed thing. Okay, thanks a lot. There’s a lot more to read about this on Foojay, of course. That’s it for this episode of the Foojay podcast. Thank you, Simon, for sharing your insights that your part of the security team helps us a lot to know what’s going on. Thank you also to my listeners for staying until the end. Go and run Java version to see how far behind your runtime is and if you didn’t update yet, go straight to the builds of October 24th and maybe by the time we release this podcast, there is a new one. We never know what happens this weekend. You find the links in the description in the show notes of this podcast. Please subscribe on YouTube or in your favorite podcast app so you don’t miss the next episode.

[43:40] Most probably with more security-related info and best practices related to AI findings and other stuff. And if you have something to share with the Java community, become a Foojay author or join us on Foojay Slack. All the info is on Foojay.io. See you in the next episode. Thank you.

[43:58] Give me a Foo. Give me a Jay. Give me the friends of OpenJDK.

Share this page

Found a mistake, or something to add? Edit this page on GitHub

Written by

Frank Delporte

Frank Delporte is a Java Champion, Java Developer, Senior Technical Writer at Azul, Blogger, Author of "Java Programming for Raspberry Pi - A Hands-On Guide to Electronics and IoT Projects", and Open-Source Contributor for Pi4J, Lottie4J, Sheetmusic4J and …

Written by

Simon Ritter

Simon is Deputy CTO at Azul.

Related posts

Join the discussion