Asking the LLM for something is like doing an "Ask the Audience" in Who Wants to be a Millionaire. You're basically polling the consensus answer to the question you're asking.
> You're basically polling the consensus answer to the question you're asking.
That's perfectly fine, because that's what you expect from human work to begin with. From natural text like blog article or tecnical reports to software changes, all output is expected to comply with patterns we are already familiar with. Heck, look at pull requests, where you ask your audience to evaluate your work hoping to reach a consensus.
You'd expect that from an average person, but not a genius in the subject. "Genius" is almost by definition a contrarian viewpoint or technique that happens to be better than the consensus or the best-known.
> You're basically polling the consensus answer to the question you're asking.
There is nothing wrong with going with the consensus answer. Sure, you can't invent an O(N) sorting algorithm with the consensus, but LLMs can definitely write good, maintainable code. Michelin star analogy might be an exaggeration, but it can help cook a good homemade meal, possibly even better than most wannabe home cooks.
>There is nothing wrong with going with the consensus answer.
For most things, yes. But you don't hire experts to tell you what everyone else knows. You hire experts to give you the angle.
For example, an LLM trained on all AWS documentation wouldn't be super useful (possibly less useful than traditional full-text search), because what you really want to know is the things that AWS didn't write, the writing "between the lines", all the things that DynamoDB can't do.
I found one of my license texts for a hobby project[0] in a Microsoft product[1]. I have no idea how they use it.
Copilot is very poor at understanding how the DSL arrangement of the project works or how to assemble validator trees. Before Copilot many other "enterprise" stuff like JetBrains IDEs had problems auto-completing it. But it was always good enough for me, for my own things.
It obviously have value. Maybe they're lazy and using it for something unrelated like the TLD list. I really don't know. What I know is that something of my amateur dish ended up being served in what I consider to be a fast food chain.
No, copilot is not endlessly variable. It works best with typed languages and an enterprise mindset. That's exactly the point against "vibe coding" they push. MS seems to despise anything that doesn't follow that.
Look, I love typescript and all that stuff. As I said, McDonalds meals are delicious.
But I don't want that approach for everything I code. This project is not the only one with that issue. I noticed the same in a portable shell compatibility layer, it can only help if I make it to look like other script languages.
"You're asking too much, how can copilot help with a language it was not trained on?"
I am not asking it to be good at that. Just don't make asshole posts saying anything other than MS pasteurization is "vibes".
If they paid me, then that would be another story. But they didn't, and used my "vibe" project anyways.
I am not complaining about software I released for free, before anyone tries to go for that.
I am making an observation about kinds of software that copilot can't help with. My kind of software development, in which I might take months to decide a name or redo everything several times just to try a new kind of sauce.
> No, copilot is not endlessly variable. It works best with typed languages and an enterprise mindset.
I don't think your assumption is true. The key factor is the corpus used to train it, and the context you fed it. I already experienced Copilot fumbling references to methods of a simple Java class, whereas it pulled off thinks like ARM templates flawlessly. You need to understand that internally LLMs do probability-based text completion. If you feed them enough context to maximize the probability they output things you expect, they do so.
Right, but that's exactly what static typing does in most cases - it provides model with more explicit context for what it's doing.
For the same reason, they tend to handle XML better than JSON - sure, both are trees, but XML is has redundancy, and that redundancy helps keep the model on the rails, so to speak.
I actually wonder if the perfect LLM language would be something a lot more like COBOL - not in a sense of being similarly high level, but rather verbosity. And perhaps also being closer to natural English, which is, after all, still a lot of its training set, especially for reasoning stuff. For query languages they seem to like SQL the most of all the things I've tried, and I strongly suspect that it's the same underlying cause.
Of course, a new language designed like that would have the fundamental problem that there's no existing corpus in such a language. Then again, if it is also designed such that one can reliably convert e.g. from Java to that language, then perhaps we can still pull that off.
> For query languages, at least, they seem to like SQL the most of all the things I've tried, and I strongly suspect that it's the same underlying cause.
The reason they like SQL more than other query languages is primarily that their training data has orders of magnitude more of it than any other query language, and that advantage is so huge that any other possible advantage would probably have comparatively negligible effect.
I'm not so sure about that. This was in comparison to e.g. walking object graphs in Python, C#, and JavaScript using the usual APIs (i.e. where querying one one-to-many relationship looks like Foo.Bars rather than joins, and using map/fold/filter or equivalents). I would expect there to be a lot more of that kinda stuff in the training set than SQL.
