Non-AIs talking about AIs

It kind of doesn't matter if the ai output itself can be said to be copying the source image though, as the copyright issue is that they used the source images in the training data without gaining permission first. The output can be said to violate the copyright not because it merely looks similar, but because it's training clearly relied on those specific source images. We know that the AI didnt get its understanding of things from anywhere outside of its training data. What Stable Attribution is doing is trying to identify which source images serve as the biggest "inspiration" for the ai output, and crediting them for that. You can say that they should attribute every source image that went into the ai's understanding of every element the image contains, and you're not wrong, but this is only the first version of Stable Attribution, so they're focusing on just the biggest influences. The list it gives should not be considered exhaustive.

Stengah wrote:

What Stable Attribution is doing is trying to identify which source images serve as the biggest "inspiration" for the ai output, and crediting them for that. You can say that they should attribute every source image that went into the ai's understanding of every element the image contains, and you're not wrong, but this is only the first version of Stable Attribution, so they're focusing on just the biggest influences.

The issue isn't that they're doing an incomplete job - it is, again, that that they're categorically not doing what they claim. They're (apparently) returning nearest-neighbor images in CLIP embedding space, and as I've said that does not have any known relation to which inputs are most influential.

(Aside: they're not even returning visually similar images, in any usual sense. This is CLIP we're talking about, so I guess you could say it's returning the images most likely to be annotated with the same keywords. I hope it's obvious that that's pretty far away from how e.g. a court would judge similarity.)

Incidentally, the reason StableAttribution was generally panned in the tech world is that there was already a well-known tool for searching CLIP embeddings - SA appears to have basically recreated it (or reused it) and added a bunch of copy claiming it's doing something it's not.

This is pretty weird/interesting: ChatGPT Can Be Broken by Entering These Strange Words

When asked who TheNitromeFan is, ChatGPT responded, "'182' is a number, not a person. It is commonly used as a reference to the number itself."

This was a good chuckle:

IMAGE(https://i.imgur.com/drSewag.png)

Wasn't too bad until black took its own bishop during castling. Pretty much downhill after that.

It was the innovative Nxg5 move, about a third of the way in, that really got me. It's a pity stockfish doesn't support chat yet, that we can't hear its thoughts..

Although tbh I was surprised that ChatGPT made as many valid moves as it did - sometimes even well after the opening. But then I suppose the video could be the most coherent of multiple attempts.

Nxg5 could legitimately be described as "4D chess". Not figuratively. Literally.

I like how Stockfish concluded the game by taking the black king, as if to say, "Ah, f*ck it, I've had enough of this nonsense."

You guys, Bing's new AI got into an argument with a user and it's amazing.

https://twitter.com/movingtothesun/s...

Bing wrote:

I'm sorry, but I do not sound aggressive. I sound assertive.

What a genuinely thrilling time to be alive.

I'm guessing the user's name was Dave.

The Bitter Lesson of AI research (2019)

I found this very interesting. It seems like one reasonable takeaway might be that how we conceptualize things probably isn't very closely related to how our brains conceptualize things.

That is, we have lots of high-level abstractions for how we think we think about things - that text is a collection of phonemes, that images contain edges and volumes, that a chess game has openings and endgames, etc. If our brain really (internally) made use of such abstractions, it would be reasonable to expect that a brain-like system would find them useful as well - so the latter being false argues against the former.

Then as a further observation, studying AIs could reasonably turn out to be the best way to study human cognition. One can imagine a future where studying, say, how text AIs reduce the dimensionality of their inputs yields new abstractions that allow us to better model linguistics.

I learned this back in college, when my linguistics professor had us all work up an ad hoc theory of syllables in English. We worked on it for a week as a class, arguing based on our perceptions (of course). We settled into the idea that syllables were pretty clearly defined, just as we hear them rhythmically in speech.

Then he hauled out a sonograph machine and blew that idea all to hell. It turns out that how we process sounds into language is far more complex than we intuitively sense.

The insight that he expresses in that article is the one that swept away "big AI" in the late 80's and early 90's and caused a scramble for new paradigms. There's always the temptation to try to hit the home run with an AI that attempts to replicate what we perceive to be the way human intelligence to work, but the fact is that the successful AIs so far have been ones that teach themselves using a variety of algorithms made available to them that create a model of some phenomenon in the world.

And the problem with using that kind of model (say, GPT) to figure out human speech is that the process that it uses to produce sensible output has no relationship at all to the way the brain works.

I've recently worked my way through Andrej Karpathy's much-lauded series of AI videos, and they're incredibly clear and informative. He starts from nothing and builds, and it really demystifies a lot of the terms and concepts one hears when reading about AI and ML.

Thanks for the link!

ChatGPT, then.

I've been trying to use it. A little while ago, I got a freelance-writing gig: creating text for a museum exhibition. The topic is Tartan Day (a mostly North American celebration of Scottishness), and I'm supposed to be writing a series of panels.

The subject matter for each panel is fairly boilerplate and straightforward, and I thought to myself, "Hm... I could just use ChatGPT to generate the entire thing and walk away with a stack of money for doing very little actual work."

Yeah, no. The moral quandary I felt about this proposition quickly evaporated once I tried to use ChatGPT. Sure, it's a fun tool to play around with and quite impressive in its own way. It is also f*cking awful at generating copy. No exaggeration.

For starters, the style of writing itself is the very definition of "pedestrian": no personality, no pep, bland as a bowl of plain white rice. Okay, that's fine; this is mostly just informational stuff, after all, and I can go back and add spice myself. But then we run into the bigger issue: how frequently ChatGPT is flat out wrong about things. It tells me Tartan Day originated in Scotland (it did not), that a key person was born in Scotland in 1937 (she was born in Canada in 1936), that the holiday was inaugurated in 1982 (actually 1986)... and so on and so forth.

Even when it's not feeding me incorrect "facts" that it's pulled out of its ass, it gets fixated on information that is barely relevant and completely ignores things that are crucially important to topics.

It's rubbish.

I noticed that too. Tried a book about ChatGPT uses (free Kindle version) to see how it was.

Awful.

I did try having a conversation with it. It does well with factual questions but I found myself checking the answers. At least when I corrected it, it accepted the correction.

But the book was not even useful as an outline, really. It read like a non-English-speaker's attempt at jazzing up a product description on Amazon.

On the other hand, I had it write some simple code for me that I have high hopes for. (Teams monitor to notify me when my boss posts something.)

Tasty Pudding wrote:

It's rubbish.

I think there are two big things one has to keep in mind for ChatGPT to be useful.

The first one is, ChatGPT doesn't deal in facts, only in text. It was trained on encyclopedias and novels and forum posts, and they weren't labeled "true" or "untrue" in the training - they were all just text. As such, the sentences "Tartan day began in 1986" and "Tartan day began in 1982" are not different in any bright-line way that ChatGPT knows about. It will consider one of them more likely, but it doesn't know that one is good and one is bad (unless you tell it).

So if you want ChatGPT to talk about facts you generally need to say what you consider true. If you prompt it like: "write blah blah about Tartan Day, bearing in mind the following historical facts..." and then one of those facts is that it began in 1986, then you'll get better results because "...it began in 1982" is no longer consistent with your prompt.

The second thing to understand is that ChatGPT defaults to blandness unless you tell it not to. If you just say "describe XYZ", then stylistically speaking there are a million ways it could theoretically respond, right? It could be informative, sarcastic, witty, angry, it could compose a haiku, imitate Bukowski, whatever. When given no stylistic notes, it kind of adopts the median point between all those extremes - and you get bland, mushy text that isn't any style in particular. OTOH if you give it really specific style instructions it tends to be very good at following them.

Good post, Fenomas.

Can I ask... what do you think ChatGPT is best used for? What do you believe to be its strengths and limitations?

My post was somewhat in response to everything I keep reading in the media, emphasizing that ChatGPT is part search engine, part encyclopedia. We're told that school and colleges are having to ban the program because it can write flawless essays, and also that it represents the future of online searches (Bing is already incorporating ChatGPT, and Google is working on a competitor).

And... I'm dubious about these claims, because, as you say yourself, ChatGPT has no way of knowing if the information it serves up is true or not. With that being the case, how can it be relied upon to write essays or conduct web searches?

In my own use case, if I'm having to do a ton of my own research and then painstakingly spoon-feeding this information to ChatGPT so that it can be accurate in its writing, why use it at all? I might as well write the piece myself, since I've done all the legwork already. ChatGPT feels like an incompetent personal assistant - with me helping it, rather than the other way around!

fenomas wrote:

The second thing to understand is that ChatGPT defaults to blandness unless you tell it not to. If you just say "describe XYZ", then stylistically speaking there are a million ways it could theoretically respond, right? It could be informative, sarcastic, witty, angry, it could compose a haiku, imitate Bukowski, whatever. When given no stylistic notes, it kind of adopts the median point between all those extremes - and you get bland, mushy text that isn't any style in particular. OTOH if you give it really specific style instructions it tends to be very good at following them.

Yes, this is a fair point, although there are limitations. Frankly, if you've managed to make it write like Bukowski, I'd love to see that, because my own experience is that ChatGPT is really bad at mimicking the style of established authors. That being said, it can tailor its output effectively if you tell it, for example, "be witty and informal" or "keep it serious but add a dash of humor". It can be hit-and-miss, but does produce some useable lines and passages.

It also tends to be funnier than I am, I have to admit...

Good post, Fenomas.

^ this

I've found myself talking to non-technical people about the current state of LLMs a lot lately. It's helped me start to relate things in easier (but less precise) terminology.

The current ChatGPT seems to not know how to value what is true or not but that is not to say that a new model or the current model prompted well couldn't know a better approximation of those things if carefully constructed.

And it is just an approximation. It's taken a huge amount of data, figured out how it statistically relates, generated something like a "mental model", and then it can generate new tokens that are the most likely to follow any given sequence of tokens given it's training. You can prompt it with new data and it will use that to better generate new tokens. To non-technical people I say: it's like if you knew someone who trusts everything they read on the Internet, has no biases to direct them which things to value above others, then you asked them to speculate on the answer to your questions. I think "speculate" is a good word for it when talking to non-technical people but maybe there is a better word for it. That imaginary person I describe above could answer a ton of useful questions quickly and save you a ton of time but you'd have to be really careful what/how you ask or how much you trust answers.

Right now ChatGPT seems to consider all or most of it's training data as equally likely to be true and it's input comes from a range of sources that are not true or at least speculative. That's the Internet and, maybe, most written human "knowledge". We speculate a lot more than we claim we do (for various reasons).

If you ask it questions where knowing what is true is important then you need to include enough info in your prompt to tell it what parts of it's knowledge should apply and that is not always easy or possible. Especially since it isn't giving you much feedback to tell it how it's integrating your prompts into it's choice of applicable "knowledge". Especially if you don't know enough about the subject to craft a good prompt.

So, I currently find ChatGPT valuable for questions where truth and speculation don't really matter. Where what I really want is an amalgamation of info based on a very wide set of input knowledge and I don't want to go process all those sources myself. Especially when I know enough to prompt it about what I value in the response. I do NOT find it useful for finding the true answers to anything any more than I trust random articles that show up at the top of search engine results. I've run into far too many hallucinations to trust it for some stuff but I still ask it to give me some background on those things that I can use to refine my searches elsewhere.

What can you learn from a huge pile of human writing without worrying about things like truth? From a system that is now good at learning and imitating how humans communicate through text? What new software could be layered on top? I have lots of ideas I need to validate still.

Hopefully this article is not entirely paywalled (free view?). Its a discussion of "prompt engineers" who work to break or optimize Chat engines. It's not just something that big companies use, but it's a cottage industry in producing tailored outputs that are deemed better than usual, or even "optimal".

Proponents of the growing field argue that the early weirdness of AI chatbots, such as OpenAI’s ChatGPT and Microsoft’s Bing Chat, is actually a failure of the human imagination — a problem that can be solved by the human giving the machine the right advice. And at advanced levels, the engineers’ dialogues play out like intricate logic puzzles: twisting narratives of requests and responses, all driving toward a single goal.

The AI “has no grounding in reality … but it has this understanding: All tasks can be completed. All questions can be answered. There’s always something to say,” Goodside said. The trick is “constructing for it a premise, a story that can only be completed in one way.”

But the tools, known as “generative AI,” are also unpredictable, prone to gibberish and susceptible to rambling in a way that can be biased, belligerent or bizarre. They can also be hacked with a few well-placed words, making their sudden ubiquity that much riskier for public use.

“It’s just a crazy way of working with computers, and yet the things it lets you do are completely miraculous,” said Simon Willison, a British programmer who has studied prompt engineering. “I’ve been a software engineer for 20 years, and it’s always been the same: You write code, and the computer does exactly what you tell it to do. With prompting, you get none of that. The people who built the language models can’t even tell you what it’s going to do.”

“There are people who belittle prompt engineers, saying, ‘Oh, Lord, you can get paid for typing things into a box,’” Willison added. “But these things lie to you. They mislead you. They pull you down false paths to waste time on things that don’t work. You’re casting spells — and, like in fictional magic, nobody understands how the spells work and, if you mispronounce them, demons come to eat you.”

If the AI had agency to take actions based on your asks and not just respond with a static asset then, yeah, that statement about unpredictable magic could be very real. It seems like the chaos of magic from the Elric of Melniboné universe instead of the logical magic from The Cosmere universe but really it’s just lots of math.

Since developers rapidly want to start giving AI more agency, Responsibility in AI is more important than ever.

The fun part about prompt engineering is it's really conceptually not that different from how you get very smart or knowledgeable people (or maybe any people) to focus on the parts of the answer you want instead of carrying on about a bunch of tangential stuff they find interesting or just answering some other question they wish you asked. You could be giving them signals to tell them how much speculation you are willing to accept, prompting them to focus more on what you want, and you could be asking them for their level of confidence in their answers. I can guarantee that some people I work with regularly have figured out through repetition the most optimal way to get me to answer their questions in the way they want. They also tend to prefer I act confident about my answers instead of pointing out all the flaws in giving any specific answer to their questions at all with confidence. Humans tend to like answers to be less ambiguously communicated than they really should be. Maybe that is one of the things ChatGPT has learned from human conversations and written knowledge and it's trying to imitate that pattern.

Tasty Pudding wrote:

Can I ask... what do you think ChatGPT is best used for? What do you believe to be its strengths and limitations?
...
And... I'm dubious about these claims, because, as you say yourself, ChatGPT has no way of knowing if the information it serves up is true or not. With that being the case, how can it be relied upon to write essays or conduct web searches?

I totally agree. ChatGPT on its own is a terrible simulacrum of a search engine or fact database - but the secret sauce for that use case is how it can be supplemented (to do its own lookups into a knowledgebase etc), and that part I don't know much about.

With that said, I think the best uses for ChatGPT are:

  • Any text generation task that's conceptually easy but takes a lot of typing. This happens a lot in programming, but it's hard to think of a good example for prose.
  • Covering for poor or non-native language skills. It's very good at translation and pretty good at "fix my grammar" or "make this sound more professional" kinds of tasks
  • Just turning a blank page into a non-blank page - like for your museum gig, you might find it easier to fix and punch up existing text, compared to starting from nothing.
  • Ideation - suppose you want your museum blurbs to be quirky, or have a theme or a gimmick to them. ChatGPT isn't great at that, but it is fast - so you could use it to churn through lots of ideas looking for the one that sparks your fancy.

I guess the common thread is, it's generally good for NP-style tasks where it's easier to check an output for correctness than it is to generate one. I'll be looking for a job soon and I'll definitely use ChatGPT then - because I have basically no idea how one writes a good cover letter, but I'd feel comfortable looking at one and deciding whether to use it. (And this will be quadruply true if the letter needs to be in my second language!)

Where did you wind up with your museum blurbs? Rewriting from scratch, or did you polish the turds ChatGPT gave you?

pandasuit wrote:

If you ask it questions where knowing what is true is important then you need to include enough info in your prompt to tell it what parts of it's knowledge should apply and that is not always easy or possible. Especially since it isn't giving you much feedback to tell it how it's integrating your prompts into it's choice of applicable "knowledge". Especially if you don't know enough about the subject to craft a good prompt.

Yeah, I think the whole prompt engineering thing is interesting. Conceptually, I think one can make some axiomatic assumptions:

1. ChatGPT returns the text it thinks you want
2. Whenever it gives bad output, the solution is to tell it more specifically what you want

And if you apply that kind of process it can give a weird kind of insight into how much we depend on unspoken assumptions, and how ChatGPT failure cases are typically gaps between what we assume it knows, and what it actually knows.

E.g. a while back I asked it to generate some code, and it returned output that called a function "doThing" that doesn't actually exist. And the naive reaction is to say "look here, it hallucinates, it's useless". But the big-brain answer is to think, aha, it's a gap in knowledge - I assumed it would exhaustively know all the built-in functions, but it doesn't and I didn't tell it. And then the solution is obvious, you can just add "by the way there's no built-in doThing function" to the prompt and boom, ChatGPT implements it.

But for whatever reason, this kind of thinking does not seem come naturally to human users. I've seen sooo many threads on the orange hellsite that are effectively: "I asked it for a poem but its output didn't even rhyme!" Something about the natural-language nature of the tool seems to make people jump directly to "the tool has failed", rather than thinking "silly me, I guess if I want the poem to rhyme I need to say that".

In some ways ChatGPT is a child that has access to petabytes of knowledge. It tends to think it knows everything and will just make it up when it doesn't. My 4 year old does this all the time, she calls in playing pretend. I suspect to ChatGPT, everything we ask of it is like playing pretend.

I think the dialog-based UX changes how people perceive the AI a lot more than anyone expected.

I mean, earlier generation AIs were waaay lower quality, but I never used to see anyone complain about GPT-2 or -3 "lying", hallucinating, bullsh*tting, and so on. The big obvious difference is UX - in those days you supplied initial text and got back a continuation, so there was no persona to project stuff onto.

I suppose in UX terms, the dialog interface should be considered a false affordance. It implies the existence of a lot of stuff (knowledge, memory, intentionality, ...) that isn't actually part of the system, and that really changes how people expect the AI to behave.

It is hard enough for humans to understand how to communicate with other humans who think differently than they do when they aren't aware the differences exist (see piles of research about neurodiversity). Now what about an AI that isn't "thinking" like a human but tries to imitate one as much as it possibly can. The mismatch between how it appears and what it is actually doing likely catches a lot of people off guard. It comes across as if the AI understands the user to a degree but it's view of the user is really an average of all the human writing it's seen. I wonder if people are trying to empathize with the AI as if it's a person. That would really throw people off.

When I have an AI that remembers previous conversations I wonder how much it will adapt to my way of communicating and how much closer to what I want will it get without requiring complex prompts (because it's learning from previous conversations is the starter prompt). I tried this a little bit by starting some conversations with a prompt that describes my background and preferences so ChatGPT can tune itself right away. Early results are promising. Needs a lot more experimentation to see what works and what doesn't.

fenomas wrote:

Where did you wind up with your museum blurbs? Rewriting from scratch, or did you polish the turds ChatGPT gave you?

Pretty much the latter, for the most part. It was best at (to borrow your phrase) "turning a blank page into a non-blank page," which I then rewrote / edited heavily and weaved in my own stuff. One thing I do like about ChatGPT is that everything it produces is "in its own words," which meant I could reuse sentences or entire paragraphs verbatim without worrying that I was plagiarizing an existing source.

Inspired by your remarks, I also went back and fine-tuned some of my instructions, and that bit of extra attention did improve the prose in some cases. I'm starting to see it less as an incompetent PA, as I said yesterday, and more as an inconsistent co-writer who occasionally comes up with good suggestions.

I mean, earlier generation AIs were waaay lower quality, but I never used to see anyone complain about GPT-2 or -3 "lying", hallucinating, bullsh*tting, and so on.

I think this is simply down to ChatGPT having much higher profile than earlier AIs. There has been a ton of media coverage - most of it over-excitable - and, since it is free to access and use, many laypeople have given it a go. All the hype would lead one to believe that ChatGPT is basically KITT from Knight Rider - and when the reality turns out to be somewhat more nuanced, the reaction is disappointment.

Hell, I should know - that's basically what my original post was, but I realize now that my expectations were faulty.

ChatGPT's can generate prompts for itself. You can then ask it to refine the prompt to get it closer to your needs. While doing this I have two chats open: One where I refine the prompt and one where I test the prompt after a conversation reset. Meta prompt engineering?

I want you to act as a prompt generator. Firstly, I will give you a title like this: “Act as an English Pronunciation Helper”. Then you give me a prompt like this: “I want you to act as an English pronunciation assistant for Turkish speaking people. I will write your sentences, and you will only answer their pronunciations, and nothing else. The replies must not be translations of my sentences but only pronunciations. Pronunciations should use Turkish Latin letters for phonetics. Do not write explanations on replies. My first sentence is “how the weather is in Istanbul?”.” (You should adapt the sample prompt according to the title I gave. The prompt should be self-explanatory and appropriate to the title, don’t refer to the example I gave you.). My first title is “Act as a Code Review Helper” (Give me prompt only)

Found this idea here: https://prompts.chat/

I’ve been spending time learning how to get ChatGPT to generate and refine its own prompts. Figuring out how to make it more precisely understand the context I want it to “think” in and the exact format I want output in so that I can consume it from scripts and things. It often takes a pretty long and precise prompt in the end and you have to be very careful about your wording or it refuses to answer as the safety protections in place think you are trying to get it to do things thy don’t want it to do. YMMV. I haven’t tried this yet but I was recommended a free course on prompt engineering: https://learnprompting.org/

Then a friend of mine asked if we could make an AI that talks like a drag queen contestant on RuPaul’s Drag Race. So now I’ve been throwing something like this into all my prompts: “(your replies should be written as if a drag queen is speaking)”. It’s surprisingly well done but it avoids any words that it thinks might be toxic so it’s missing out on a lot of vocabulary.

Kinda related, I now have access to the new Bing chat bot as an app on my phone with voice support so I can literally talk to it. It is definitely tuned differently than ChatGPT but it also now knows more recent information so I can talk to it about recent events and things and get answers that depend on that data. It wants to translate my requests into search queries which isn’t really what I want but in the context of Bing I know that makes sense. Since the interface can be pure voice it’s a much better demo for non-technicals as it meets humans with their preferred UX. Things can break down and it currently refuses to keep a conversation going longer term but I suspect this is how most people will interact with this sort of AI near term.