My recent experience with generative AI chatbots

General Discussion
21

I do not understand why AI chatbots would do that?
Are they trying to make it look less robotic?
They need to inject some creativeness rather than randomness.

3 Likes
22

Here is the AI response to which I was referring:

Yes—there is typically a stochastic (random) element in responses from models like me.

Why randomness is used

When generating text, the model predicts a probability distribution over possible next tokens (words or pieces of words). Instead of always choosing the single most probable token, the system often samples from that distribution. This introduces variability and helps produce:

  • More natural language

  • Diverse responses

  • Less repetitive phrasing

This process is related to settings such as temperature, top-p (nucleus sampling), and other sampling parameters used in the decoding stage of language models.

Deterministic vs stochastic behavior

  • Deterministic mode: If the system always picks the highest-probability token (effectively temperature = 0), then the same prompt can produce the same output assuming everything else is identical.

  • Stochastic mode (typical): With temperature > 0 or sampling enabled, the same prompt can produce different responses across runs.

Other sources of variation

Even when randomness is reduced, responses might still differ due to:

  • Model updates or version changes

  • System prompts or hidden instructions

  • Context differences

  • Numerical nondeterminism in large GPU computations

In this chat interface

You should assume:

  • Responses may vary slightly if you repeat the same prompt.

  • But they are usually similar in meaning and structure.

If you’re curious, I can also explain exactly where randomness enters the token generation pipeline and why even temperature = 0 doesn’t always guarantee perfect reproducibility in large inference systems.

5 Likes
23

Are they saying that large GPU calculations run out of of significant digits even with 64 bit floating point numbers?

OK, I can see how randomness is introduced.
It is a sort of artificial variety of speech … vary tbe presentation, but hopefully not the meaning.

Thanks Don.

3 Likes