To be perfectly honest, Clojure's biggest mistake, from my point of view, was to be so heavily JVM-dependent.
Yes, in certain environments Java is king and a ton of people use it, it's battle-proven, etc. But so are other stacks.
Clojure, as a programming language, sparks a lot of joy and is a lot of fun to use and reason about. Yet in reality, when working in a production environment, you're basically fighting the always-memory-hungry JVM.
Just give me Clojure running on BEAM and I'll take it over anything in this world.
Not really. Few if any platforms compete with the JVM on performance and observability (monitoring/management) and ecosystem. It's technologically head and shoulders above pretty much all other runtimes (e.g. BEAM has excellent observability, but loses big time on performance and ecosystem). Also, if you're OK with the JVM running at BEAM speeds, you can give it far less memory.
.NET has come up on par, or slightly better than, the JVM in recent years in terms of performance. Ecosystem may not be as big but its pretty well fleshed out.
.NET is the only runtime that comes close. Unfortunately, some early design decisions (reified generics) make language interop on the CLR much more cumbersome than on the JVM. Clojure in particular makes heavy use of the fact that a Clojure map/list/set is a Java map/list/set. This is awkward on the CLR (where you must specialize the collection).
Initial implementation of Clojure targeted CLR and JVM. Urban legend says: Rich Hickey named the language "Clojure" because it has 'c', 'l', 'r' and 'j' in it.
I have found that quite hard to get working in various aspects (from runtime installation to package management and compatibility with the rest of the ecosystem).
.NET-the-technology is pretty good, but for many developers, .NET is synonymous with expensive licensing and tooling (and you pretty much have to use Windows, ick).
The JVM and .NET are presently about the same capability-wise, but Java is (or at least, is perceived as) much easier and cheaper to freely experiment with. Unless you need to use C# for some deep Windows integration, this makes Java the obvious choice for anyone without sunk costs in the .NET ecosystem.
The JVM doesn't use more memory than other virtual machines, the problem is that it doesn't release memory back to the operating system, which is why it is being configured with a maximum heap size. Also compacting garbage collectors need more memory available for de-fragmenting the heap.
I've been running a simple JVM web app on my own personal server with an allocated 128 MB of RAM and a lot of that is actually unused. You can do a lot with a heap of just 64 MB.
People aren't aware, but when running your average PHP app you need more than that with just a couple of requests done at the same time, whereas my JVM server can easily handle 1000 requests per second. Similarly Ruby or Python apps can be big resource hogs, especially when you end up setting up multiple parallel processes.
I was also surprised to find that compilation to "native" isn't necessarily efficient in terms of used memory. I've built a similar app in Haskell and the used memory ended up being higher and more unpredictable. If something like Go fares better, it's not because it is compiled to native.
All in all the JVM is actually quite efficient in terms of used memory, for a garbage collected language ;-)
Considering Hetzner (hetzner.com) offer VPSs with 2Gb RAM for 2.99 Euros per month I think it's fair to say we've left the 90s behind as far as RAM is concerned.
You get an OOM if your working set is larger than the heap. Otherwise, the GC just works harder, reducing your performance. Generally, the JVM uses as much memory as you let it.
JVM has two new open source garbage collector implementations that are designed for TERABYTES of heap space with 10 ms GC pause times. (yes you read that correctly)
ZGC from Oracle -- but open source
Shenandoah from Red Hat -- also open source
I know of no other managed runtime platform with the GC options and tunability as JVM.
Also JVM has not one, but TWO JIT compilers, C1 and C2. Your JVM byte code starts out interpreted. It is constantly dynamically performance profiled. If it is getting above average CPU use then it is quickly compiled by C1 into native code, and scheduled to be recompiled soon later by C2. When C2 comes along, it takes its time recompiling your code into highly optimized native code.
Suppose that YOUR function A calls MY function B. And C2 aggressively inlined my B function into your A function. Now suppose my code is dynamically reloaded within the running environment. Now your function has a stale version of my code inlined within. JVM will instantly de-optimize your code back to JVM byte code interpretation so that soon, if it is still a cpu hot spot, it will get recompiled again by C1 and C2.
Let me know of any other runtime platform that even comes close to the things JVM does.
It isn't. Perhaps you haven't seen what the JVM can do lately. It's got JIT compilers that do this[1], GCs that are getting close to 1ms max pause times on terabytes of heap [2], and low-overhead, in-production deep profiling [3]. Not to mention all of the classic capabilities of the JVM like hot code redefinition and exceptional monitoring. Lots of applications can certainly manage well without some or even all those things, and some want other tradeoffs (like no warmup or lower RAM footprint), but these days it's hard to compete with the JVM where it's strong. The technological gap between it and other runtimes is only growing.
This is the dumbest thing in the universe (in context of a Clojure discussion) that I keep seeing parroted over and over. So you have to type "recur <args>" instead of the actual function name + <args> for the exact same effect. If you can get over that aesthetic "sin" then you have your TCO in effect. I don't even find it mildly inconvenient...and even prefer the "shortcut" syntax vs explicitly repeating my function name.
My point: if someone is going to assert that another platform is competitive with JVM, then not only is performance important, but capabilities as well.
While no language / platform / os / etc is perfect, JVM as a managed runtime platform is hard to beat. More than two decades of research in its JIT compilers, and multiple GC implementations with various knobs and dials for tuning and monitoring. Battle tested.
Someone mentioned SBCL. While I am a fan of Common Lisp, and generally all Lisps, I would cringe to see SBCL with terabytes of heap and hundreds of CPU cores running a serious workload -- and see how well it holds up.
Just an example: You can Google for this, but in 2012 Twitter switched from Ruby on Rails to Java. They have YouTube videos explaining their change over. Basic reasons: performance and scalability. They have to handle BILLIONS of tweets per day and route each one to multiple places with notifications in near real time.
> I would cringe to see SBCL with terabytes of heap and hundreds of CPU cores running a serious workload
That's nothing SBCL can do, but one can save an executable and start that in a subsecond without the need for a terabyte of RAM.
Generally there is a lot of stuff in the JVM which makes it less attractive as runtime for Lisp: more complex code loading via 'class loaders', lack of support for memory representation of Lisp's data types (like CLOS objects, which are more dynamic than Java classes), lack of TCO, lack of easy AOT compilation, lack of easy image dumps, lack of resumeable exceptions, ...
And in 2019 something like Eclipse still is a bit clunky to run on my 8 core Xeon - part of the reason could be a less than great GUI implementation. In IntelliJ I need to restart my IDE for every simple plugin... probably features like live-updating haven't made it yet into popular apps.
> but one can save an executable and start that in a subsecond without the need for a terabyte of RAM.
A few tens of magabytes maybe and the JVM runs a complete Hello World in about 60ms, but yeah, Clojure does a lot of work on startup. It's more of a Clojure issue than a JVM issue, though. It's true that HotSpot is mostly designed for long-running applications. Other JVMs -- like Substrate VM (AKA Graal native image, which isn't quite a JVM yet) and Excelsior JET give you AOT compilation (no warmup).
> makes it less attractive as runtime for Lisp
I think Clojurists (which probably outnumber all other professional Lispers combined several times over) would disagree. I'm sure there are use cases where you'd like to use Lisp and HotSpot is not the right choice, but it seems like those cases are vastly outnumbered by the cases where it is. Domains where HotSpot is certainly not the right choice are usually either client-side web or domains where C/C++ dominate.
HotSpot's monitoring, management and observability are almost unmatched (maybe Smalltalk is somewhat better in some regards, and BEAM in some others). Whatever you may be lacking on that front is probably offset by something probably even more valuable (these capabilities have been designed over the past decades to suit the needs of millions of developers running huge backbones).
I'm not saying that HotSpot is always the best tool for the job, but it's the best tool for the job for a huge portion of the software industry.
> And in 2019 something like Eclipse still is a bit clunky to run on my 8 core Xeon - part of the reason could be a less than great GUI implementation. In IntelliJ I need to restart my IDE for every simple plugin... probably features like live-updating haven't made it yet into popular apps.
You don't need to use a Java IDE if you want to write Clojure, although the Java IDEs are also pretty unmatched -- a matter of getting used to.
So I need to switch to exotic JVMs to get better performance?
> I think Clojurists (which probably outnumber all other professional Lispers combined several times over) would disagree
Does the number matter, since most of them have a) never used Lisp nor b) do they know anything about Lisp implementation? Things like 'no TCO' in the JVM are no question of the number of people using it, it's simply a fact.
If the JVM would be a better match, then Lisp implementations on it would have a better start-up time - like most Lisp runtimes have.
There are many reasons to use the JVM, but good support in the JVM for Lisp implementations is not one of the strong points.
> Delimited continuations and eventually tail calls are coming as part of Project Loom
Which means, after reading the linked doc, that neither resumeable exceptions nor TCO are in JVM, neither currently nor in the near future. Now the usual arguments about TCO are: a) one does not need TCO b) it makes tracing difficult c) it's a security problem d) we can do recursive self-calls e) a future version of the JVM will provide annotatations for tail calls f) using TCO would provide interop problems with JVM code... etc etc.
The fact remains: TCO is not supported by the JVM, while this capability is a part of many other runtimes.
> probably even more valuable (these capabilities have been designed over the past decades to suit the needs of millions of developers running huge backbones).
That's all great, but I don't develop huge backbones all the time.
> I'm not saying that HotSpot is always the best tool for the job
Your original claim was this: 'if someone is going to assert that another platform is competitive with JVM, then not only is performance important, but capabilities as well.'
We are now talking about capabilities like full TCO and the JVM simply does not provide it. It currently provides no TCO at all.
> You don't need to use a Java IDE
It's not about what I need, it's that capabilities like extending the IDE on the fly are still not available in an IDE like IntelliJ - why? After downloading a plugin, it wants to restart the IDE. Something which a typical Lisp system on a Lisp runtime brings for free, because of its capabilities.
> So I need to switch to exotic JVMs to get better performance?
Depends what you mean by "better". HotSpot is optimized for peak performance (which is higher than that of other Lisp runtimes), and to suit the needs of server-side development. Other VMs target other uses -- fast startup, embedded etc.
> If the JVM would be a better match, then Lisp implementations on it would have a better start-up time - like most Lisp runtimes have.
It's already a better match, because that's what most Lispers prefer. If you mean it could be even better than it is, then I agree. I also agree that it's not the best match for, say, Racket. Now, it's perfectly fine if you prefer Racket over Clojure, but most people prefer Clojure to Racket, partly because overall the runtime gives them more of what's most important to them. Would it be cool if the JVM had more features that would allow it to be a better match for Racket? Sure.
> Does the number matter
Yes, because Clojure is the most popular Lisp, the features it has are probably the ones most important to Lispers.
> Things like 'no TCO' in the JVM are no question of the number of people using it, it's simply a fact.
Yes, but how much not having TCO matters can be answered by popularity. That people use Clojure much more than they use Lisps on runtimes with TCO suggests that other things matter to them more.
> neither currently nor in the near future
Currently, no. Near future -- probably. I'm the technical lead for that project, and it's very likely that delimited continuations would arrive soon (in fact, you can already build the prototype today, and early access builds will be available in a couple of months).
> Now the usual arguments about TCO are ...
Well, as it's being addressed only now rather than, say, 10 years ago means that it's not a top priority, but judged to justify some reasonable effort at this point in time.
> TCO is not supported by the JVM, while this capability is a part of many other runtimes.
True, but the fact remains that the totality of capabilities of the JVM is considered more attractive by more people than that of most other runtimes.
> That's all great, but I don't develop huge backbones all the time.
I am not saying that you, personally, should definitely use the JVM, only that it seems that the JVM is a great platform for many languages, including Lisps, and that the use cases it targets are very desired.
> Your original claim was this ...
I believe I also said that I mean capabilities for the domain HotSpot targets. Obviously, other things may be more attractive for, say, embedded software.
> It currently provides no TCO at all.
There are other things it doesn't provide. But by comparing capabilities I did not mean one by one, but the totality of those of one runtime vs. those of the others.
> It's not about what I need, it's that capabilities like extending the IDE on the fly are still not available in an IDE like IntelliJ - why? After downloading a plugin, it wants to restart the IDE. Something which a typical Lisp system on a Lisp runtime brings for free, because of its capabilities.
No, that capability is definitely there. I'm guessing the IDE authors just didn't think it's worthwhile to use. NetBeans, for example, doesn't require a restart after downloading plugins (or, rather, rarely does).
> I'm the technical lead for that project, and we're very likely to have continuations soon.
What relationship have continuations to making Java exceptions resumeable?
> But by comparing capabilities I did not mean one by one, but the totality of those of one runtime vs. those of the others.
Not sure how you compare a totality, other than actually having a list and compare them one by one.
My feeling: you make a claim, but you have never used one of the Lisp runtimes like Allegro CL and you simply don't know what they provide and when it comes to specific capabilities, they are not important to YOU, because JVM users don't use them - given that they have no choice.
The JVM isn't the predominant enterprise platform because it is the best Lisp platform. Very few people in companies care whether it runs Lisp well or not. People have chosen it as their platform for their niche language, because of its domination in the specific enterprise market - not because it is technically or capability-wise a good platform for their niche language. Actually even though the JVM now exists for a long time, none of the dynamic languages have their best / most performant / dominant implementations on the JVM. Not Smalltalk, not Lisp, not Scheme, not Python, not Ruby, ...
> What relationship have continuations to making Java exceptions resumeable?
We don't want resumable exceptions for Java. But if you want to implement a Lisp that has them on the JVM, you'll be able to use continuations for that.
> but you have never used one of the Lisp runtimes like Allegro CL and you simply don't know what they provide and when it comes to specific capabilities, they are not important to YOU, because JVM users don't use them - given that they have no choice.
I have never used Allegro, and, as I said, some capabilities that are important to some people may be missing on the JVM. But it seems that most Lispers prefer what the JVM has to offer, which is high performance, low-latency GC, low-overhead monitoring, and a huge ecosystem.
> not because it is technically or capability-wise a good platform for their niche language.
True, but most people don't care about what's good for their language, they care about what's good for their program (and users). It seems like more people prefer Lisp on the JVM, even if they have to give up some features (and they'll need to give up fewer features with Loom).
> Not Smalltalk, not Lisp, not Scheme, not Python, not Ruby, ...
Python and Ruby are actually most performant on the JVM nowadays. Look up TruffleRuby and GraalPython. They're just very new. Even TruffleJS is almost as good as V8.
So the Java libs I'd have to use would not support them. Not so great.
> I have never used Allegro
So the totality of your opinion of the JVM being better for Lisp comes from exactly what knowledge? Oracle marketing?
> But it seems that most Lispers prefer what the JVM has to offer, which is high performance, low-latency GC, low-overhead monitoring, and a huge ecosystem.
You mean Clojure users recruited from Java? I have never seen many Scheme, Common Lisp, Javascript, Julia, ... users on the JVM.
> Python and Ruby are actually most performant on the JVM nowadays. Look up TruffleRuby and GraalPython. They're just very new. Even TruffleJS is almost as good as V8.
Really? Looks to me that it's not particular fast.
[GraalVM CE, Java 1.8.0_202] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Please note: This Python implementation is in the very early stages,
and can run little more than basic benchmarks at this point.
Well, resumable exceptions are not so great... We really don't want them in Java (remember that Java is a language originally designed by Lispers).
> So the totality of your opinion of the JVM being better for Lisp comes from exactly what knowledge? Oracle marketing?
I've used Racket (and PLT Scheme before it) quite a bit (I was a Schemer when Java barely even existed), and Guile, and, of course, Clojure. Also, Oracle marketing is quite terrible.
> You mean Clojure users recruited from Java? I have never seen many Scheme, Common Lisp, Javascript, Julia, ... users on the JVM.
I started using Scheme in the mid-nineties. Unfortunately, I haven't seen many Scheme and Common Lisp users anywhere. Clojure is the first in over three decades to be used on any noticeable scale in industry. There are plenty of people that run JS on the JVM (probably ten times all non-Clojure Lispers), and Julia is still new.
> Really? Looks to me that it's not particular fast.
> We really don't want them in Java (remember that Java is a language originally designed by Lispers).
Who of the Oak team were Lispers? Not even Gosling has ever seen or used a competent Lisp implementation - he wrote a 'Mock' Lisp without lists ;-) for an Emacs written in C on Unix. One of his many projects. Stallman later implemented a slightly better Lisp using Goslings C code as a start.
> I've used Racket (and PLT Scheme before it) quite a bit (I was a Schemer when Java barely even existed), and Guile
None of them have especially interesting runtimes. Racket now gets a new runtime based on Chez Scheme from Cisco. Chez Scheme is actually great. You should have studied that. Well, source code is now available from Cisco. It has a nice AOT native compiler.
> I started using Scheme in the mid-nineties. Unfortunately, I haven't seen many Scheme and Common Lisp users anywhere
You were too late. SUN had its own Lisp offerings in the 80s. Like DEC/DIGITAL, IBM, Apple, HP, Apollo, Texas Instruments, Xerox, ...
Yes, I know Chez Scheme, and I know about the excitement around Lisp in the 80s, and now Clojure is giving Lisp a bit of a resurgence. I assure you that Lisp still has fans in the Java team today. I also know about various cool features in various runtimes -- we monitor the field closely and learn from everyone, but as those who've built the most successful programming runtime in the history of computing, which is also one of the most successful and capable Lisp runtimes in the history of computing, and also made not one but several breakthroughs in JIT compilation, garbage collection and low-overhead monitoring, the Java team knows a thing or two, as well, especially about which features have a big impact.
Java outperforms SBCL by a very wide margin -- 1.25-4x faster -- except in the short-running benchmarks (and except the spectral-norm test, where SBCL is 5% faster), as the game does not perform warmup, and includes Java's compilation time (you can see that no Java result is under 2 seconds).
I don't know of any other SBCL benchmarks, so, as I said, it's probably better than nothing.
I have looked at some example programs in the shootout game where different implementations use completely different low-level algorithms, including code that's very unidiomatic, so those benchmarks are of low quality. And yes, Java does have a disadvantage here (I work on OpenJDK, BTW, so I don't need to imagine).
Anyway, I’m not sure which side you’re arguing, because the benchmark game did show a rather drastic win to Java (TBH, I would have been surprised if that were not the case). I simply qualified that win because the shootout game is not a high quality benchmark.
Alright, so first of all, to qualify my "name calling" let me say this: the benchmark game does have a lot of valuable data, that can be useful for all sorts of things. In particular, it can be used to study which low-level algorithms and language specific optimizations make a difference when comparing different programs in the same language. Of all the things the data can be used for, one of the least useful ones -- to the point of being nearly meaningless -- is comparing the performance of different languages to each other, the thing that the game's presentation emphasizes most. Why is it nearly useless for that? Because the programs were written by different people and use different low-level algorithms; sometimes the benchmarks measure compiler quality, sometimes library quality, and most often it measures the optimization skill (and patience) of the various program authors. Also, usually comparisons are made along one axis -- same program, different compilers; different languages but same algorithm and author; different authors but same language and compiler; here, all factors change with every test. Here, we have two programs presented next to each other, with differences along three or four variables, each can impact the result by tens or even hundreds of percent, and then we see a difference of, say, 40% between the two. What does that even mean?
All these things make the inter-language comparison as presented nearly useless. Crucially, the site itself doesn't even state what it is that it purports to measure or compare, like a paper without any hypotheses, claims or exposition. That alone is enough to place it in the low-quality benchmark category (regardless of the merits of the problems themselves, which I haven't looked at). But that's where most non-professional benchamrks are. ¯\_(ツ)_/¯
Now, I would guess that there is some meta-hypothesis to the game pertaining to markets, but that's not what's being analyzed or even presented.
> Which programs?
I tried to look at programs that report big performace differences and I'm looking at C++'s regex-redux, and I see it uses a library that isn't in the standard library. Or look even at "Java" vs "Substrate VM" pidigits. These are two VMs for the same language, yet the programs are completely different. I'm not sure what I could learn from that about HotSpot vs Substrate VM.
Or, the example that caught my eye when I reached the game when looking for SBCL benchmarks -- SBCL's reverse-complement, that is reported to outperform C by more than 10x. Well, not only does it use a completely different low level algorithm (doesn't use parallelism), it doesn't outperform C or Java at all, but crashes, and what appears to be some junk result is reported. Had it not crashed, it would still have used a very different algorithm.
> So why haven't you told us how few 1/10ths of a second?
Because that would be misleading. Now, I admit that what threw me off at first was Java's performance compared to SBCL's reverse-complement, but that's just a mistake and has nothing to do with compilation. Real programs can take many seconds to compile, but looking at the benchmarks, they do look quite small, so you are correct that they will be compiled rather quickly (less than a second). However, I see that Java is run with tiered compilation, and obtaining good optimization, even for such small programs, can still take many seconds (say 10 or 30). As a general rule of thumb, we always warmup for at least a few seconds, even for microbenchmarks.
It is true that the difference would often not be large compared to the inter-language differences reported by the benchmark -- and as I said above, it's unclear what it is they compare -- but they can certainly be in the 5-10% range, and maybe more. This is a big difference for what I call "high quality" benchmarks, benchmarks intended to really give a precise performance measurement. For example, Twitter has a whole machine learning system that tunes and re-tunes their JVMs just so they can get a 7% performance improvement. Even the game's own examples that you linked to and seem to say, see, warmup doesn't matter, report differences of 10-200%.
> Which programs?
Now I'm looking at Haskell's fannkuch-redux
> And don't you expect code written to be fast to be different?
Yes, and if the comparison was for the same language (and compiler) then this would be interesting, but what does this tell you when comparing different languages?
> > So when you say "the relative speed of those languages" you actually mean the relative speed of programs?
> No, I mean the relative speed of programs with some carefully chosen relation between them. For example, there is a big difference between "relative income of individuals" and "relative income of social groups," even though the latter could be reduced to the former. Whether a comparison of individuals is meaningful as a comparison of groups depends on how the individuals are chosen and aggregated.
You can choose which program measurements to compare.
> > We can make mystery for ourselves — no one can force us to understand.
> True, but when a website chooses to present data in some aggregate way that seems to defy any familiar forms of aggregation and reasoning without so much as stating what the meaning of the presentation is, it is itself mysterious.
You have not shown that is what that website does.
I agree that it is possible that the comparisons are not worthless, but without any statement of a hypothesis, it is the most reasonable conclusion.
Yet another tendentious assertion!
Program X measured less than Program Y.
I am sure that the authors of the website would agree that if they don't even attempt to say what the meaning of the comparisons is, even in their own minds, it is reasonable to assume it means nothing.
Once more, you are putting your thoughts and words in others' mouths.
> You can choose which program measurements to compare.
Yes, but the website also chooses, and it is the website's choice that I think is bad. Also, it doesn't seem like the game even has comparable programs in some/many/most cases.
> You have not shown that is what that website does.
How can I show what a website doesn't do? If you think they say what the hypothesis is, show me. I haven't been able to find it.
> Program X measured less than Program Y.
First of all, when something means nothing other than itself, we say it has no meaning. If the comparisons on the website are meant to suggest nothing more than just their own definition, then they are meaningless.
Also -- come on. The website presents comparisons for specific choices of X and Y. What is the meaning of the choice and the resulting comparison? Your position that it's not meant to mean anything (other than itself), sounds disingenuous because the website still makes some very specific choices that seem to hint at a meaning (e.g. why not compare the second fastest programs between languages instead of the fastest?) yet the meaning is never explained.
You can try to show your claim — "… chooses to present data in some aggregate way that seems to defy any familiar forms of aggregation and reasoning…" — has some basis in reality.
> … when something means nothing other than itself, we say it has no meaning.
That may be what you say. Show why we should think that's what people interested in "Why did Clojure gain so much popularity?" would say.
> … why not compare the second fastest programs between languages instead of the fastest?
Inexplicable! Or obvious: there would need to be two programs for each, rather than one.
> Anyway, we're going in circles.
I agree, you repeat your assertions without establishing anything.
1) Of all the things the data can be used for, one of the least useful ones -- to the point of being nearly meaningless -- is comparing the performance of different languages to each other, the thing that the game's presentation emphasizes most.
Is that what the presentation emphasizes most?
"Will your toy benchmark program be faster if you write it in a different programming language? It depends how you write it!"
"Which programs are faster?"
"Which programs are fast?"
"Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages."
2) Or look even at "Java" vs "Substrate VM" pidigits.
Currently 2 Java pidigits programs are shown.
The program with the fastest measurements on OpenJDK uses GMP and something different needs to be done to make that program work on Substrate VM.
3) …but crashes, and what appears to be some junk result is reported.
What is reported? Measurements at the largest workload both programs worked.
4) examples … you… seem to say, see, warmup doesn't matter
See, for programs with runtimes of 0.24s, 1/10th of a second is a lot!
See, for programs with runtimes of 4s and 20s, 1/10ths of a second… !
5) Now I'm looking at Haskell's fannkuch-redux
And… ?
6) … if the comparison was for the same language (and compiler) then this would be interesting, but what does this tell you when comparing different languages?
That seems to be a subjective statement about your interests.
Someone else might think that neither "tell you" much until you investigate Why?
Yes, because that's what the site shows most prominently: comparisons between languages.
The fact that the explanations to the benchmarks basically say that these comparisons mean nothing doesn't make the comparisons mean much more than nothing, especially as there's no explanation to what the comparisons do (or are supposed to) mean. So in addition to having no hypothesis, no analysis and no exposition, there's a notice saying, don't use these comparisons for so and so. Then what are those comparisons for?
Again, I don't think the game's comparisons are worse than many other benchmark comparisons out there, but that doesn't make it good. What's unfortunate is that unlike those other benchmarks, they have a lot of data that could be used for some really cool stuff (not to compare languages).
It's possible that the language competition is meant to encourage more people to submit and show how their favorite language wins, but there's no information about the market aspects.
> See, for programs with runtimes of 4s and 20s, 1/10ths of a second… !
I run a toy benchmark that's the same size as the game's, and HotSpot's C2 gives me a 30-50% boost after ~10s. It really varies by program (not by problem). This tends to be worse for programs that use a lot of parallelism and aren't configured to run compilation in blocking mode, because then the application threads compete with the compilation threads, but again, there isn't really a good rule of thumb here. In any event, it is simply wrong that C2 would never yield significant performance boosts after a few tenths of a second, even on small programs (it may be true for some specific benchmark entries and maybe even all current ones, it may vary by JDK version, and it definitely varies on whether HotSpot uses Graal or C2). Although, given all the other serious problems that render the comparisons mostly meaningless (different algorithms), my guess would be that this effect is smaller by comparison.
Still, there's a lot of cool stuff the game could do with the current programs (and historical ones), but comparing languages is not really one of them.
> … but there's no information about the market aspects.
What do you mean by "market aspects" ?
> …it may be true for some specific benchmark entries and maybe even all current ones…
So it would be wrong to try to dismiss those measurements, based on nothing more than extrapolation about how some other toy benchmark program might perform.
> … cool stuff the game could do …
You're allowed your own personal opinion about what's cool and what isn't (and no doubt others will have their own personal opinion).
It's certainly not obvious to me what they comparisons mean (is it obvious to you?) and the authors of the site don't even state, let alone support, a hypothesis. They just say what the comparisons don't mean. So unless it's a work of art, I think observers can be forgiven for thinking it doesn't mean anything.
> What do you mean by "market aspects"?
I mean a hypothesis about "free market competition" driving people to write really fast programs.
> You're allowed your own personal opinion about what's cool and what isn't
True, but I meant something that could be used in a more rigorous/meaningful way (e.g. comparisons where only one variable changes). Meaning is perhaps also subjective, but I don't even know the subjective opinion of the meaning behind the game, because the site doesn't say.
Anyway, now that I've taken a slightly closer look, I realize I spoke too soon when I said that the SBCL vs. Java benchmark is "better than nothing." If, say, Java uses parallelism while SBCL doesn't, then if Java is 4x faster that may not be a "better than nothing" indication.
Seems like your idea of a comparison, between for-example Lisp and Java programs, would require writing all the Lisp programs as if they were being written in Java. (We used up our "one variable change" with SBCL instead-of OpenJDK).
> I spoke too soon
You chose to summarize data: "Java outperforms SBCL by a very wide margin -- 1.25-4x faster".
Do you mean that you made mistakes when you made your summary.
That's like asking what does a program do and being told, "it does what the code says." Technically true, but not what we mean when we ask. Also, I don't think that's really what the site's authors thought. After all, when I click on the Haskell page, I don't see a comparison of Haskell programs, but a comparison of the fastest Haskell program with the fastest Java program. It's not obvious to me what that means, or even that it means anything at all.
> Seems like your idea of a comparison, between for-example Lisp and Java programs, would require writing all the Lisp programs as if they were being written in Java. (We used up our "one variable change" with SBCL instead-of OpenJDK).
That's exactly right, but because that's rather tedious, I would present just meaningful comparisons rather than non-meaningful ones, so no inter-language comparisons. I'm not saying they're always meaningless, but they can't be made with the data currently available on that site. Unfortunately, we cannot manufacture (non-fictitious) meaning out of data as we like.
> Do you mean that you made mistakes when you made your summary.
Yes. By "better than nothing" I overestimated the significance of results, at least in this case. After a closer look I realized that different programs use different algorithms, and so the comparison tells me nothing about the relative speed of those languages.
No it is not. Program X may not have measured less than Program Y.
> … meaningful comparisons…
Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right?
1) You seem to be contradicting your own argument.
2) Your desire for "rigorous" comparison is in-conflict with your desire for inter-language comparison.
> … different programs use different algorithms…
For some programs, for some definition of "algorithm".
> … the comparison tells me nothing about the relative speed of those languages.
> Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right?
I may have misinterpreted your comment. By "as if they were written in Java" I meant using a similar algorithm. Alternatively, there could be a rule about idiomatic code (enforced by experts), which would compare very different things, but would also be at least somewhat meaningful.
> For some programs
I don't know. Whenever someone refers me to a result on that site that looks weird, I see very different algorithms.
> for some definition of "algorithm".
For a very well-accepted definition of an algorithm in the context of real performance comparisons.
> What is the speed of a programming language?
I'm not sure (although perhaps I could come up with some definition given enough thought), but I didn't make a website that compares languages based on performance. In fact, I said that the website is totally meaningless. But the people behind it may have something in mind, and if they do, they're not telling us. All I can do is think of ways to use the site's data in ways that can at least be meaningful.
Will someone else look at those similar algorithms and "see very different algorithms"?
> … a result on that site that looks weird, I see very different algorithms.
Do regex libraries for different languages in-fact always implement the same algorithms?
> What is the speed of a programming language? I'm not sure…
You did say "Java outperforms SBCL by a very wide margin -- 1.25-4x faster".
Saying "… so the comparison tells me nothing about the relative speed of those languages" suggests you are looking for something that tells us about "the relative speed" of those languages?
> Will someone else look at those similar algorithms and "see very different algorithms"?
I would think so.
> Do regex libraries for different languages in-fact always implement the same algorithms?
I don't know. But I do know that when something is presented as an answer, it's good to know what the question is.
> suggests you are looking for something that tells us about "the relative speed" of those languages?
I can think of a couple of ways the two can compare. One is the relative speeds (and other metrics) of programs of a similar nature, written simple idiomatic code. This gives a general "feel" for what kind of workloads different languages may be suitable for. Another is comparing the performance of the compiler/interpreter on some microbenchmark of the same algorithms, preferably written by people who coordinate, and preferably without mixing too many variables (e.g. measuring parallel scheduling separately and regex separately). Either one of these is of some interest.
The benchmark game does neither. If there is some idea behind the site's comparisons, it's certainly keeping it a mystery.
Browsing some more I see that the binary-trees programs for Java and Substrate -- again, same language -- use completely different algorithms. Not to mention that just the "Java" programs alone use such a weird range of parallelism constructs, that I have no idea whether there was any reason behind it, the programmers simply didn't know about proper parallelism constructs, or the program is so old that it was written before newer constructs were introduced.
I understand that, but it still makes the comparison between the two pretty useless. Those are literally two compilers for the same language. I see a comparison of two compilers compiling two very different programs. What does that mean?
As someone who has done some Clojure, including some smaller projects in production at a big company, Clojure running on BEAM would have been a complete non-starter.
It's really easy to introduce Clojure into a Java shop, because if things go south, you can throw away a bit of code and replace it with Java. Heck, people on other teams don't even have to know this started as Clojure code.
Asking people to learn both a new way of developing and a new VM? Too much all at once, which is why Clojure spread as far as quickly as it did.
> To be perfectly honest, Clojure's biggest mistake, from my point of view, was to be so heavily JVM-dependent.
With Clojure being a hosted language, there's nothing stopping it from adopting a new platform or expanding to other platforms or languages. It has actually already done this with Clojurescript (run in the browser or Node.js)[1] and Clojure CLR (run on Microsoft's .NET)[2].
I think this is accurate (in terms of exposure, not that it's hardly used). As soon as it was possible for other languages to run on the JVM, the huge amount of developers working around JVM infrastructure that didn't like the sometimes painful experience of working with Java, started looking at alternatives.
Groovy, Scala and Clojure were the early contenders that people were exploring and Clojure seems to have won a lot of mindshare.
The thing about Java isn't the language or the JVM it's the App Servers. Enterprises have invested heavily in the App Server infrastructure for deploying JVM based code from monitoring, training, etc.
Throwing all that away to use a non-JVM language is a REALLY hard sell. Those app servers are very well done.
Running on the JVM made using Clojure, Scala and Groovy possible in places where other languages wouldn't have been. In the early 2000s, it was Java and everything else.
The .NET ecosystem was awful at that time. The programming world was essentially, Java, Perl and PHP 4 in terms of enterprise usage. People said the words "Enterprise Java Beans" without a chuckle.
People were HUNGRY for better options.
Now you've got a solid C# ecosystem, Elixir, Go, a dominant Python ecosystem, significantly more polished PHP, K8's & Docker that make automating the infrastructure around all different languages consistent...it's just a different world.
In the early 2000s, it was Java and everything else.
It still kind of is in a lot of places. My country barely bothers to even teach anything else at a lot of schools, and enterprise is almost entirely dominated by the JVM.
You are exactly right. Having worked in Clojure and talked to a lot of other devs here and across Europe, most everyone got into Clojure or Scala because management and cultural complaints left them stuck with the JVM, and at least it beat building more EnterpriseWidgetFactoryGeneratorPatterns.
Oracle and Java are the new IBM, and just like IBM, no one's really happy with them but management yells if you use anything else.
>Groovy, Scala and Clojure were the early contenders that people were exploring and Clojure seems to have won.
I love Clojure and hope it gains more mindshare, but I don't think this is accurate. Scala currently enjoys quite a lot more industry use than Clojure does, for instance there are currently roughly 15x as many postings mentioning Scala as there are Clojure (source: Linkedin.ca).
Same in London. Search indeed.co.uk for "title:Clojure" and there are on average no more than 6 unique results. Outside London the rest of the UK shows next to nothing. About 6 years ago Clojure dojos and meetups were well-attended but even then the numbers were a fraction of the equivalent Scala events. Nowadays you're lucky to get a handful attendees at London Clojure events which is a real shame as it suggests Clojure is disappearing into a very small niche.
> As soon as it was possible for other languages to run on the JVM, the huge amount of developers working around JVM [...] started looking at alternatives
This is true, but...
> Groovy, Scala and Clojure were the early contenders that people were exploring
... Jython and BeanShell were much earlier than those three.
Jython, JRuby, Rhino, and Clojure are JVM languages that are syntactic copies of existing non-JVM languages, and it seems JVM developers didn't want these. Clojure has won within that group.
But the other group is what developers really wanted, i.e. languages with a Java-like syntax but with extra features added. Beanshell came first with dynamic typing. Then Apache Groovy cloned Beanshell and added closures, and Beanshell never kept up. Scala had all that but also had inferred static typing, which Groovy later added but was too late to the game to have much impact. Kotlin then came and merged the best features of Scala and Groovy, and became successful on Android.
Used like "someone used their pet language for small side project" or used like "these companies would crush if Clojure compiler suddenly stopped working tomorrow"?
A lot of credit for that goes to Rich Hickey - from what I can tell, he did a very successful job early on of 1) designing a Lisp with a unique combination of ideas and 2) publicizing it in a way that resonated with a people.
I agree, Rich has done a great job promoting and maintaining Clojure. BTW, he used to use Common Lisp. I first got to know him (via email) because of his wonderful Java <--> Common Lisp bridge.
re: ABCL: this is really a great project. I don't use ABCL very often but when I need it I am grateful to the developers.
Not just that. Clojure actually has a lot of features. It's not just "yet another Lisp on JVM". here's an excerpt from Joker page (Clojure linter written in Go):
Protocols, records, structmaps, chunked seqs, transients, tagged literals, unchecked arithmetics, primitive arrays, custom data readers, transducers, validators and watch functions for vars and atoms, hierarchies, sorted maps and sets.
Clojurescript was much easier to make because it was written in Clojure, not Java. IMO if we ever see Clojure that is hosted on other runtimes it will be written in Clojure and probably still will have to depend on JVM.
Kawa was my first Lisp / Scheme and I dug it quite a bit. I wrote some wrappers for Processing called ProKawa which probably doesn't work with the latest Processing releases, but it was all very fun. Quil on Clojure is pretty good, and in general I'd say I've gotten more milage out of Clojure for sure.
Right, I'd forgotten about that! I used it as an extension language for a Java program once. I didn't think it even had a compiler, but that was 20 years ago, and it appears to have one now.
> To be perfectly honest, Clojure's biggest mistake, from my point of view, was to be so heavily JVM-dependent.
IMHO, Clojure's use of the JVM was something of a Faustian bargain. (But net worthwhile).
Reasonable people might disagree, but there is a huge amount of value in having access to a good JIT, GC, and standard library - not to mention the ecosystem around the JVM. Keeping in mind that Clojure started out as a one-person, self-funded project, it was a very reasonable thing to borrow all of this from an existing well-established language. Maybe these things could've been reproduced, but not in the approx. two years that were initially available, and it's not at all clear that building a JIT, GC, etc. would have been the best road to making a useful impact.
From what I recall of Clojure's launch and the reactions I heard from people, the JVM aspect might've been 1/3 to 1/2 of the attraction to CL people: Java was more widely popular at the time, and Clojure said "here's our angle to get more people paid to Lisp".
My self-interested perspective on Clojure today (as an expert in a couple other Lisp niches) is that I'd be happy to learn Clojure if someone would pay me to, but it's not relatively attractive to me now, in the ways that a Lisp is more attractive than a non-Lisp. (Nor in the way that Rust is in some ways more attractive than C and C++. I've been a senior developer in those, and Rust is sufficiently different, and might open some new doors.)
If we look at it from a historical perspective, Clojure came about at the end of the Java and Ruby eras where running big long-lasting programs was the norm. It was about 2 years later that Node.js, Go, and then Rust came about. These languages specialized in faster startup times, which made them more amenable to a variety of use-cases (where Java and Ruby were mostly used as web servers).
If we just look at developments of the last 5 years, and in particular serverless, it's been about taking advantage of those short startup times to use in a server context. When you take that into account, these languages become even more in the lead.
I think the next lisp that has a chance to really take of will be one built on wasm.
MRI ruby had always had a fast startup time and was not originally created for the web... it’s more like a “better” Perl cum toolkit for C/unixy stuff- ironically what python became more popular for despite python not originally being written for that purpose.
Also in the old days of the web.. CGI was literally spawning a process per request.
Regardless it’s bizarre to think nodejs pioneered startup time... Ruby, lua, python, and good lord Perl would really like to have a word with you for starters.
Java was really more the exception.
And I’m not sure it’s true that these serverless platforms are spawning a new process per request ... but wow what is old is new again. Sounds like you’re describing inetd (look it up). Nothing of what you’re talking about is 5 years. Try 30-40.
You're misreading my comment. I'm talking about general trends in programming. The 2000s were dominated by bulky server environments like Java, IIS (for .NET) and Ruby Mongrel. These were not things you started up just by running a script.
The 2010s the trends changed towards languages that specialized in faster startup times. That led to serverless.
No one said that these are new inventions. Like all trends its cyclical. I'm just pointing out the trends in order to better understand where Clojure fits in.
No not misreading. Ruby mongrel was the antithesis of a bulky server environment, it was a single threaded worker host generally spawned behind something like Apache or nginx or lighttpd. The need to keep mongrel workers around was more a function of the slow startup of rails then ruby or mongrel itself.
Like I said the Java (and .NET) were the exception. As far as trends... java and .net dominate enterprise and they still do. The trend is very slow in the grand scheme.
Frankly your comment seems to be conflating language issues with framework issues with “serverless” and even scripting. I don’t think there is a cogent argument to be had about overall trends with all that. Yes lambda and serverless are a thing but half the whole world still runs on JavaEE or ASP behemoths
You absolutely can, and you can compile JVM Clojure to native binaries via GraalVM. For example, somebody wrapped up my Migratus library into a standalone migration tool recently https://github.com/leafclick/pgmig
It's a community maintained dialect. Its maintainer, Juan Facorro is very active. You can see him giving a talk about it here: https://youtu.be/Ow8o9Mm_N7M
I'm sure all the project is in need of are more adopters to really get it off the ground. Check it out!
I’ve played with LFE (http://lfe.io) and found it quite enjoyable, but it isn’t Clojure and requires some understanding of Erlang-and some getting used to.
I don't know if JVM is a problem; the issue that caught me in my few-weeks of experimenting with the language was the lazy semantics of sequences.
I know the problem, and I've danced around this same issue in things like Haskell with strictness annotations, and I've run into the same problem in R, but I'd like to just be able to read and parse a ~1GB CSV file without blowing out RAM with a giant list of thunks.
This was many years ago now, so maybe they've fixed it, but I'm tired of fighting that particular battle.
The biggest disadvantage of Elixir today - it doesn't have its own thing comparable with Clojurescript. Clojurescript is not just extremely nice compiler that gives you syntactic sugar, you can totally share code between JVM and JS and that's incredibly awesome.
As a Ruby dev turned Elixir dev: no. I mean, they both have `do..end` blocks, which I don't love in either language. But the similarities pretty much end there.
Yes, in certain environments Java is king and a ton of people use it, it's battle-proven, etc. But so are other stacks.
Clojure, as a programming language, sparks a lot of joy and is a lot of fun to use and reason about. Yet in reality, when working in a production environment, you're basically fighting the always-memory-hungry JVM.
Just give me Clojure running on BEAM and I'll take it over anything in this world.