I asked it something like "give me suggestions for distinguishing situation X from not-X probabilistically with data from a smartphone's sensor array".

It gives me a bunch of plausible suggestions, but the bullet points look suspiciously like an embellished list of all of the commonly used sensors on a phone. (why shouldn't it...)

So I decide to zero in on one of them arbitrarily, and I ask it to characterize the probability distribution for sensor Y, in the "X" and "not-X" case (I'm not wording it like I do here, so don't jump to conclusions)

Then it says, contradicting the earlier response, that really, you can't expect any difference between X and not-X and it avoids giving anything quantitative. It smoothly describes a distribution centered around zero with a small variance, but that's the very opposite of helpful.

Ok, but I keep reading that it's very useful if you know how to prompt it, if you break things down into smaller steps, and so on.

So next I ask it to provide me some of the competing sources for the two contradictory claims, that the output of sensor Y is, or is not, differing depending on X. If ChatGPT can't understand its sources, perhaps I can read them.

And it gives me a really nice looking result, with three citations and hyperlinks for each.

But thennnnn...I click on the links. First for the less probable claim that sensor Y can determine X. Two go nowhere (possibly related to having numbers in the URL) and the third goes to a completely unrelated study about...cats.

Ok, but I know what you want to say...you want to say it's still useful if you can see the other three links aren't garbage. Alas, it didn't work that way. None of them was verifiable, and the only one that wasn't verifiably worthless went to a very large and paywalled document, impossible to be sure about. I would bet the contents aren't even in the training data though.

Next I pointed out the invalid links supporting what I wanted to be true, that sensor Y is useful for my purpose. ChatGPT immediately groveled, as it does, saying it had reviewed all the information, and I was correct, and there was nothing on the internet supporting its original suggestion.

But that doesn't help! You can't prove a negative, and it obviously hasn't done any sort of search. I stated that, but it apparently was stuck in its local min/max, I suppose due to the prior conversation.

At this point it seems clear to me that these language models are simply ELIZA combined with a method for auto-generating language rules from data instead of a hand-written script.

But everything I read to the contrary is leading to a sense of derealization. Are the people who are writing what seems like fan-fiction trying to fool others? Or are they fooled? Or am I?

One thing seems clear to me - the tone or flavor of ChatGPT content, that seems both inhuman and annoying when trying to get anything useful out of it, does not seem new to me. A great deal of online journalism for quite some time resembles it in my opinion, take it for what you will.