Can LLMs Be Funny?

The aim of this document is basically to add in my raw thoughts, feelings or anything I discovered while attempting these assignments.

Sticking true to this tweet, my aim is to write this document without using any LLM tools or grammar/spell checkers[1] (even turned off google docs' tools for grammar and spell check). This document is just pure raw thoughts (so there might be some mistakes).

The idea was simple; I'd use our agent-orchestrator project to get resources, brief explanations of the reading materials mentioned in the assignment and get a plan to just get the "basic" template code ready with cursor code (hence the .cursorrules)[2], then everything else I'd try to do by myself (with some help of AI to troubleshoot errors and write comments). Below is the architecture I used to work on "joker-llm " (my project name for this assignment lol).

The questions which I will answer (while trying to dive deep on my project) are the following:

• Why did you pick the particular project?
• If you had more compute/time, what would you have done?
• What did you learn in the project?
• What surprised you the most?
• If you had to write a paper on the project, what else needs to be done?

---

#Why did you pick the particular project?

Let's begin with the first question. "Why?" To be honest, I wanted to attempt all three and even asked Monty if I could submit multiple assignments, but one of the main issues which held me back from submitting multiple assignments was my sheer bad luck. My lightning.ai  account was not working, so I couldn't really access the GPUs which I needed to run some of my thoughts for the neural atari-project. My idea was to try and implement something similar but for Pacman (and it would have definitely been ambitious); but system restrictions (aka being GPU poor) meant I couldn't do it completely, but I do want to try them out after I submit this assignment because I genuinely enjoyed researching and reading up materials for the assignment I chose.

The assignment which I chose was the Plansearch implementation for jokes. Having worked with LLMs; a LOT, I knew that one of the main domains which they "suck at" is in understanding situations. Jokes are something which require on-the-feet thinking, you need to be quick and witty for jokes to land. You can't have your punchline hitting after half of the people have left the hall (which did happen for some of the jokes which I generated lol). Plansearch helps with this via Observations and creating Plans; which get translated into code. Another reason I chose this is because of a psychological theory which my grandfather mentioned.

Above is Maslow's hierarchy of needs, it basically organizes human needs into a five tier pyramid system, but one of the things which "irked" me with this categorized nature of needs was that; us as humans operate at multiple levels and trying to "organize" them just results in chaos. I also realized that "humour" or "jokes" just couldn't fit in any ONE of those sections, but actually fit in all of them, depending on how you looked at it.

---

#If you had more time/compute, what would you have done?

If I had more time, I'd definitely read even more papers, like don't get me wrong, I read a LOT of papers while attempting this assignment, I think I spent more time reading materials rather than coding (which resulted in a bit of sloppy work, but I will let you judge that). I'd read more papers to understand different implementations of Plansearch, their thinking and mindset while "trying" to implement it in their systems. I'd read more material on LLM-as-a-Judge to understand bias better. Even during my interview, I was asked "what would you like to work on in the field of Responsible AI, if you came to lossfunk?" and I answered "I'd work on trying to remove biases from systems at the BASE/START; giving an example of Gemini's HILARIOUS yet very concerning image generation goof-up". I had a list of papers which were listed by agent-orchestrator (some of them discarded for irrelevancy and some added later), and out of 14 reading materials I had, I was only able to fully finish and comprehend 4 of them.

Apart from reading more papers, I'd try to fix some of the issues which I still haven't been able to solve (more of which I will talk about in the next section). One of the first issues which I faced was actually quite a silly one; since the "template" code generated by cursor was pretty barebones, I decided to test it by giving it the topic of "mumbai roads" (Parle-G of roads) and I expected some jokes, but instead I got jokes based on the topics of "programmers"; I didn't realize that the "template" code was just a hardcoded implementation of a very basic "plansearch", I used the understanding of the code and developed my own implementation which used Groq API to generate jokes and improved the planning structure a little by adding angle analysis; wherein the jokes generated would also generate the "angle" they are trying to go at. For example: a joke on "mumbai roads" should result in an angle of "their roads being horrible when hit by rain", "more potholes than driveable roads" etc,.

I was able to achieve a simple idea of this, but then I ran into a problem. The code I wrote kept hitting rate limits with Groq's API[3], but that was a quick fix by just changing the model from Qwen to Meta's Llama model.

Something more "unique" which I would want to experiment with is something which I'm also exploring with our agent-orchestrator project and that is Abductive Reasoning. You can see it in "exploring stages" in this branch .

Above is an idea I want/am exploring; instead of generating creative "angles" for the jokes, I am trying to find a way where we are defining the plan as a set of "constraints" for the joking universe. The framework is that; we try to create a "world" where we set premises like:

1. Premise 1: My dog is very smart.
2. Premise 2: I'm trying to teach my dog chess.

After these premises have been set, we try to present an observation which could be considered strange (or not thought about), something like "I'm not having much luck" then finally the banger joke hits which is still "logical" within the premises but completely unexpected; something like "The dog says I'm too predictable and he's already figured out my opening moves". An advantage of this aspect is the angle which would be generated would be something "normal" like "make a joke on the fact that dogs can't learn chess", but this premise structuring would result in more "creative" jokes. Plus they could also be novel, if tested properly.

---

#What did you learn in the project?

I learnt that LLMs SUCK at jokes, like, I knew from prior experience that they suck in understanding "situations", but very few of the "jokes" were actually "funny". The majority of them resulted in weak angles, incomplete jokes or just non-stop yapping trying to get to the punchline. Below are some examples of some jokes which were all the above stated things:

I couldn't attach all the jokes which I had (frankly I lost some of them), but above are a few, one of them is "funny", one is just "yapping" and one of them is just "incomplete". I realized that the incomplete ones were because of my architecture, but the yapping one was just LLMs spiraling into chaos. Few other observations I made:

1. When creating plans from combinatorial subsets of primitive "observations", LLMs just tend to forget previous context, now I knew that LLMs are great liars and they don't "actually" reason, but this also meant that their memorization for domainless (lesser known/trained domains like jokes) was absolutely trash. Sometimes it felt that these jokes were more of "reciting a story with a twist" rather than genuine play on words/meanings.

2. Introducing "diversity" meant that the structure had to be changed, where they needed a base environment to first understand "why are jokes funny" then they needed to learn how to curate such jokes and finally analyzing them and giving a reasoning as to "why" they are funny was a lot of work.

   1. I was able to achieve something close to this by not going the naive "judge the joke on a score of 1-10" way but instead used a method to judge the joke with different personas; wherein the judgement was based on parameters like "type of audience (casual, jolly, depressed, etc,.).

   2. I also tried to build a system to which tried to measure bias in jokes and correct it by linking it with a refinement loop based on the feedback (Critique→improve→judge→laugh). To do this, I did the following:

      1. Used Randomization: To prevent positional bias, I sent the jokes in random order in every evaluation round and also randomized their position during comparisons as well.
      2. Statistical Bias Reduction: I also calculated the correlation between the joke's position and its score to quantify how much a position is worth in score points.

---

#What surprised you the most?

Honestly, I knew that LLMs were not funny, but this just showed the gap between the current "human" comedians and machines. The lack of emotions behind jokes was pretty obvious, but there were some instances, where the jokes had a sense of "relatability", they sadly were just few of the many runs executed, but still there were times where "relatable" jokes were being generated and pushed to the top of the joking leaderboard.

Another surprising thing which I found while running evals was that there was a bias towards negative traits, by this I mean that EVEN WITH PLANSEARCH, LLMs had a hard time mapping abstract traits (like jokes or even audience persona) to numerical scales when those concepts are linked to undesirable characteristics. One of the things which I wanted to try out was "dark-jokes".

This joke by Anthony Jeselnik is a typical "dark-humour/offensive-humour" which lots of people enjoy nowadays and I realized that as MUCH as I tried for the systems to generate some "outrageous" jokes, their negative trait bias just didn't allow them to explore these type of jokes and that links to the final question.

---

#If you had to write a paper on the project, what else needs to be done?

Let's get the obvious answer out of the way, we need to prove our hypothesis which is:

To test this (in my opinion), we would need the following:

1. A baseline: Something which is currently missing in my current implementation is a simpler method/model which produces jokes. Something which, when compared with Plansearch, would show that our implementation produces better jokes than the simple model.

2. Metrics for comparison: We would need to define "what is a good joke", since the domain itself is subjective in nature, can we "quantify" it? If we "quantify" it, how would we test for "hilariousness?". Here is my approach for it, crowdsource it.

   1. The basic "most number of people who laughed at the joke", although humour is subjective, the metric for jokes being laughed at will always vary, we would be judging the humour behind the joke.

   2. With the help of crowdsourcing, we can ensure that it reaches a varied set of people who enjoy different styles of comedy. This would also push us to explore different genres of jokes.

But since the question is "if I had to write a paper, what else needs to be done?", I would explore the abductive reasoning route more, not only because its genuinely interesting but it also would judge our system on two things:

1. How premises and rules which (might) "limit" LLMs thinking and curiosity, but they could also push it explore unexplored paths for jokes, which is EXACTLY what is required for creating "funny jokes".

2. With the help of abductive reasoning, premises, rules and setting a "jokester" universe, it would also give us a bit of creative freedom with our ideas. We could explore the "dark-humour" jokes which I wanted to try.

3. This process (I assume) would also result in some novel jokes being generated, to test it we could curate a neo4j graph structure wherein we could explore the "planning, generating, evaluation and execution" process. We could even make a test suite like "JokerBench" which could be used as another form of metric.

---

I definitely had lots of fun in attempting both a) the assignments and b) writing this document answering the question. Thanks for this amazing opportunity :)

---

#Appendix

In the interest of keeping the post short, I've moved some technical notes and additional context here.