youtube i love you, but you're bringing me down
in a bastardized interpretation of fitness landscapes, i've recently been thinking about how every decision i make is a step in some direction in an infinite mountain range. in this space, my altitude isn't environmental fitness. it's something more like "personal value" or "amount self-actualized" where you can imagine standing at the peak of a hill in this endless mountain range as being highly satisfied with myself, highly engaged in my life.
some steps are uphill and take more effort. some steps are downhill and take less. sometimes it's necessary to take a step downhill to conserve energy, or to get to a different path on the mountain. but too many steps down and motherfucker, you're in a depression.
watching youtube is almost always a step downhill. that in and of itself isn't a bad thing - i really do think youtube is one of this century's triumphs - the issue is how easily youtube turns a single step into a scree run. i've had nights of binging youtube so bad that my eyes ached, and then instead of stopping, i continued to watch just with one eye open at a time. deranged.
but captivating people is youtube's job. the effectiveness of its recommendation algorithm is what determines the size of the pie that it divvies up and dishes out to an ecosystem of hundreds of thousands of content creators. better recommendations = more time on youtube = more advertisements watched = more money to go around. at no point will any of this system's structural incentives prompt someone to say "you know what? i think youtube's recommendations are too good. let's degrade them a little"1
so if youtube's addictive and unlikely to change, and i will always like to occasionally watch things on it, what is to be done?
block recommendations.
install ublock origin and add the following custom filters:
youtube.com###dismissible:matches-path(/watch) youtube.com##.ytp-endscreen-content:matches-path(/watch)
this will hide the sidebar and the thumbnails that show up at the end of the video: the website equivalent of moving the bowl of M&Ms that you have in your kitchen to a jar in the cupboard. i've been doing this for a week and the difference is extremely noticeable. i still check out my subscriptions each day and watch videos there that catch my eye, but afterwards, i just close the tab.2
stupor evaded, i can consider an option that might take me uphill. write, read, play music, reply to that message, go for a walk, meditate, just generally cultivate some fuckin' eudaimonia.
the nice thing about this is that i can also turn it off when i want. some integral parts of my personal development were catalysed by a youtube video i discovered hours deep in a binge, many songs i love i first heard on random youtube mixtapes. i can still make the time for that and switch off the filters temporarily, it just turns it into a conscious decision, not something that happens through my focus being sapped in the middle of something else, a process that is slowly turning us into a herd of mediocre dilettantes.
is this just an ornate elaboration of "no pain, no gain" ? yes. have a million people already written about how web 2.0 makes us unhappy, actually ? yes. is this cow thing i've got going on with the pictures ridiculous, romantic, and self-pitying ? yes
doesn't matter. if you suspect you might be watching too much youtube to get up the mountain, this is my recommendation.
the other day i had some fun with poetic justice's less serious cousin
an evil man from new york
once stole a shipment of chalk
he was eventually busted
one could say done and dusted
when the cops caught him on the sidewalk
or
a fraudulent author from wales
faked sales with no paper trail
she was eventually booked
when the law had a look
at her very incriminating email
Say I have a CSV with two columns:
country, users China, 113 Nicaragua, 29 Jordan, 89
and I want to aggregate these by continent. What's an easy way to do that?
Well I found this country-by-continent dataset by samayo:
[
{
"Country": "Afghanistan",
"Continent": "Asia"
},
{
"Country": "Albania",
"Continent": "Europe"
}
So I have everything I need except for a workflow. Let's evaluate some options.
I attached the two files and submitted the following prompt to ChatGPT 4o:
I have a CSV of users from each country, and a JSON file that shows which continent each country belongs to.
I would like to aggregate my users by continent, so that I can see how many users I have in total from North America, Europe, etc.
It started working on the problem immediately:
import pandas as pd users_df = pd.read_csv('/mnt/data/users.csv') continent_data = pd.read_json('/mnt/data/cbc.json')To aggregate the users by continent, I'll merge the users dataframe with the continent data based on the country and then group by the continent to get the total users for each continent.
Here is the plan:
Merge the users_df with continent_data on the country.
Group the merged data by the continent.
Aggregate the user counts.
Let's execute this.
merged_df = users_df.merge( continent_data, left_on='Country', right_on='country', how='left' ) aggregated_users_by_continent = merged_df .groupby('continent')['Users'] .sum() .reset_index() import ace_tools as tools; tools.display_dataframe_to_user( name="Aggregated Users by Continent", dataframe=aggregated_users_by_continent )
continent,Users Africa,23940 Antarctica,833 Asia,18976 Europe,22472 America,13256 Oceania,10539 South America,5300
So, if these values are correct, I'm almost certain there won't be an easier or more convenient workflow.
But that "if" is very ominous to me. How will I be able to verify its correctness once I've deskilled myself by outsourcing all of my Human Intelligence Tasks?
nushell is a shell environment that converts all stdout into structured data that can then be piped much more consistently like a functional programming language.
ls ~/Desktop | where size > 1MiB | get name ╭───┬───────────────────────────────────────────╮ │ 0 │ /Users/ikesau/Desktop/song.wav │ │ 1 │ /Users/ikesau/Desktop/ttrpg.pdf │ ╰───┴───────────────────────────────────────────╯
I love it. It balances accessibility with extensibility in a way that inspires me to understand information. I make fewer assumptions about data now that nushell makes it easy (and fun!) to examine things rigourously.
How does it handle an aggregation though? First I need to load the two tables:
let $c = nopen1 country-by-continent.json let $u = nopen users.csv
these now exist in my session's scope:
$c ╭─────┬─────────────────────┬───────────────╮ │ # │ country │ continent │ ├─────┼─────────────────────┼───────────────┤ │ 0 │ Afghanistan │ Asia │ │ 1 │ Albania │ Europe │ │ 2 │ Algeria │ Africa │
Now we can join them on Country-country2 and group them:
$u
| join --left $c Country country
| group-by --to-table continent
| each3 {
insert sum4 {
get items.Users | math sum5
}
| select group sum6
}
╭───┬───────────────┬───────╮ │ # │ group │ sum │ ├───┼───────────────┼───────┤ │ 0 │ Asia │ 18976 │ │ 1 │ Europe │ 22472 │ │ 2 │ Africa │ 23940 │ │ 3 │ Oceania │ 10539 │ │ 4 │ North America │ 13256 │ │ 5 │ Antarctica │ 833 │ │ 6 │ South America │ 5300 │ ╰───┴───────────────┴───────╯
Okay, not bad. Working out how to sum nested values after grouping them took
ages - a lot of describe and reading the docs on which
data types are compatible with which functions.7
I think with a bit more practice, I would get a lot faster at writing the pipes correctly, but I'm not a huge fan of polluting my shell history with dozens of failed attempts in the meantime, and editing commands in the terminal is a pain.
It's worth mentioning that nushell also has a polars plugin which feels like a great tool for the job.8 This is a popular usecase for dataframes so finding the correct syntax was simple.
$u | join --left $c Country country | polars into-df | polars group-by continent | polars agg [(polars col Users | polars sum)] | polars collect
╭───┬───────────────┬───────────────╮ │ # │ continent │ Users │ ├───┼───────────────┼───────────────┤ │ 0 │ Antarctica │ 833 │ │ 1 │ South America │ 5300 │ │ 2 │ Europe │ 22472 │ │ 3 │ Africa │ 23940 │ │ 4 │ Asia │ 18976 │ │ 5 │ Oceania │ 10539 │ │ 6 │ │ 19175 │ │ 7 │ North America │ 13256 │ ╰───┴───────────────┴───────────────╯
Aha! I was wondering if any of these methods would show the cases where we can't map the country to a continent - presumably that's what that 19175 is - despite the left join in the vanilla nushell method, it didn't catch this issue.
So while we're here, let's see if we can list which countries aren't being mapped. I don't think we need a dataframe for this.
$u | join --left $c Country country | where continent == null | get Country9 | to text
Türkiye Taiwan Czechia Libya Myanmar (Burma) Trinidad & Tobago Bosnia & Herzegovina Côte d’Ivoire Kosovo Fiji Congo - Kinshasa Eswatini Isle of Man Réunion Jersey Guernsey British Virgin Islands Curaçao (not set) St. Kitts & Nevis Timor-Leste Sint Maarten Congo - Brazzaville St. Helena Turks & Caicos Islands St. Vincent & Grenadines Antigua & Barbuda St. Lucia U.S. Virgin Islands São Tomé & Príncipe Falkland Islands (Islas Malvinas) Caribbean Netherlands Micronesia Åland Islands St. Martin St. Pierre & Miquelon Vatican City St. Barthélemy Svalbard & Jan Mayen U.S. Outlying Islands Wallis & Futuna
There are some very populous countries here! This is exactly the sort of problem with relying on ChatGPT: your ability to trust it depends on your ability to verify it, which is only developed through experience.
DuckDB is a portable SQL database client. You can run it as a transient SQL session, or use it as an engine to run SQL scripts. I went with the latter because I prefer to write SQL in a text editor.
Here's my scratch.sql file leveraging some cool DuckDB magic that
can work with inline references to CSV and JSON files, which I can run with
cat scratch.sql | duckdb10
SELECT
SUM(u.Users),
c.continent
FROM
'users.csv' u
LEFT JOIN 'country-by-continent.json' c
ON u.Country = c.country
GROUP BY
c.continent;
┌───────────────┬──────────────┐ │ continent │ sum(u.Users) │ │ varchar │ int128 │ ├───────────────┼──────────────┤ │ Europe │ 22472 │ │ South America │ 5300 │ │ Antarctica │ 833 │ │ Oceania │ 10539 │ │ North America │ 13256 │ │ Africa │ 23940 │ │ Asia │ 18976 │ │ │ 19175 │ └───────────────┴──────────────┘
This is about as easy as it can get, and I like that it's basically just plain SQL, which is never a waste of time to practise.
Observable is a JavaScript notebook platform with a host of features for interatively visualizing and sharing data. There are a dozen ways I could do this aggregation with Observable, including creating whole databases, but I'm going to go with the most straightforward: attaching files and writing some code.
Here's a link to the notebook.
The gist of the workflow was:
Unfortunately, as far as I can tell, Observable's SQL cell doesn't allow you to query from two tables at once11, so either you have to join your tables before you upload them, or join them with JavaScript and query that third object.
It's all quite a lot of process for answering simple questions, but it makes sharing your findings (and visualizing them) incredibly easy. Simon Wilson (the creator of datasette) uses notebooks all the time to create quick, low-friction tools. While I might not reach for them for the next question as simple as this, I really like Observable as a platform and I hope they're able to keep doing what they're doing.
At this point, the only reason I'd consider reaching for another tool is to keep it local and/or to have Copilot support.
Bun allows me to run a TypeScript REPL incredibly quickly. All I have to do is create an `index.ts` file somewhere.
bun run --watch index.ts
Now in my Copilot-supporting text editor of choice I can write up a quick script in TypeScript.
import fs from "fs/promises"
import { parse } from "csv-parse"
async function main() {
const csv: { Country: string; Users: string }[] = await fs
.readFile("users.csv", "utf-8")
.then((data) => parse(data, { columns: true, cast: true }).toArray());
const json = await fs
.readFile("country-by-continent.json", "utf-8")
.then(JSON.parse)
.then((countries: { country: string; continent: string }[]) =>
countries.reduce((acc, country) => {
acc[country.country] = country.continent;
return acc;
}, {} as { [country: string]: string })
);
const byContinent = csv.reduce((acc, { Country, Users }) => {
const continent = json[Country];
acc[continent] = (acc[continent] || 0) + parseInt(Users);
return acc;
}, {} as Record>string, number<);
console.log("byContinent", byContinent);
}
main();
byContinent {
"North America": 13256,
Europe: 22472,
Asia: 18976,
Oceania: 10539,
Africa: 23940,
"South America": 5300,
unknown: 19175,
Antarctica: 833
}
I thought this option was going to be more pleasant! Even with Copilot, a code editor, and a CSV parsing library, manually implementing a grouping algorithm is a chore.
Still. When I need a REPL to test code with, using bun this way is very good.
I think DuckDB wins? Each technique has its strengths, but writing SQL with Copilot, with automatic JSON/CSV handling is about as practical and quick as it gets.
yu-gi-oh was so smart. transformers had already proven you could forcibly extract money from parents by pretending your advertisements for children's toys comprised a television show, but hasbro still had to pay for injection moulding. what if they just printed the money instead?
in the yu-gi-oh tv show1, the main thing that all the characters do is play the card game yu-gi-oh2. all disagreements are resolved via dueling3, and even though this is shown to favour the rich, at the end of most episodes the child protagonists justly win and you feel an urge to go to the toy store and purchase yu-gi-oh cards.
the show was transitional, a ratcheting down of my unlimited childhood imagination to something more grounded in the real world. not that it was completely tethered to reality. beyond the porn-levels of plot to justify the action, in the beginning, the writers of the show didn't really understand the rules of the game - the early duels feature flagrant rules violations and completely hallucinated mechanics. nevertheless, i loved it. the writers were right in one sense: the rules were unimportant4
to illustrate this, my childhood best friend and i had sheets of cardstock that we would make our own cards with. Shadow Dragon, Night Dragon, Dark Dragon, Lightning Dragon, Dark Lightning Dragon. i wish i had them still. we didn't make any spell cards. all that mattered was that your cards had cool (dragon) names and drawings (of dragons) and that you didn't have too many cards that were stronger than anything in the show. that was another implicit rule that actually mattered - dramatic suspense requires unpredictability. even as eight-year olds we understood that our make-believe yu-gi-oh would make no sense if we were completely overpowered.
and so the show got us all the way through the funnel but wasn't able to convert. to us, it wasn't where the cards came from or if we'd paid for them. it was if they were fun to play with.
skip forward 8 years and i'm in high school playing magic the gathering with my friends. magic was the inspiration for yu-gi-oh, released in 1993, the first trading card game of its kind. by now the game has a pro scene, a janky official computer game, and a thriving secondary market where people buy individual cards to construct decks with. blah blah. you can just read this new york times article on the history of magic if you care about this stuff.5
my four friends and i were buying cheap cards online. we didn't have much money, and shipping from the U.S. was expensive. this balanced our group, because no one could afford the really powerful cards.6. we each had one deck until that got old then someone bought a new one and we'd play all the matchup permutations we could until it got boring enough for the cycle to repeat. over two years i spent maybe $300 doing this. eventually i moved and sold my cards to a game store for $50. i stopped thinking about magic much at all.
but now it's now. i've been thinking about magic again. i pitched the idea to my girlfriend, we got some introductory decks, and we had a hell of a time! it's a fun game! then the call of new cards began. familiar with where it led, i started to contemplate the alternative: all the cards are available as jpegs online. why not just use a home printer and scissors for the cards i want? a little budget cutting could save me a lot of money!
why doesn't everyone do this?
i searched the internet a little to try and find an answer, but didn't see anyone explain the complete incentive structure as i've come to understand it.
it helps to understand that proxy magic is a mix of forgery and piracy: forgery denotes that the value of the object comes from people believing it to be authentic, whereas piracy is about stealing7 a thing that has instrumental value. this isn't an airtight distinction, but it explains the psychology of, say, branded clothes quite well.8 a large part of the value of having a shirt with a brand logo on it is to show people that you can afford the shirt with the logo on it. and if you're caught wearing a fake, you need to be called out by the people who've paid the full price because otherwise you'll dilute the esteem they get.
the thing is, i don't give a SHIT about what people think about how much paper i have tied up in my cardboard - i'm all about that instrumental value. i just want to play magic.
but like with branded clothes, it's in many people's interests to not let me do this: magic is a trading card game. new cards enter circulation is through gambling on packs of 15 cards and trading them.910 gambling is addictive and people get hooked on pulling rare mythics, rationalized by the lie that they're developing a portfolio which will appreciate. this in turn, fuels a speculative secondary economy in which local game stores invest in inventory to try and make profit on the fluctuations of card prices.11
all of this depends on us agreeing that we need to pay money for magic cards, because upstream is the fact that hasbro needs to pay mark rosewater. if he stops getting paid, he'll stop designing new cards and then there'll be no new releases, no hype cycle, no profit margin for local game stores, no magic journalists, no sponsorship deals from companies that make playmats, no tournaments to fill event centres. if people stop cracking packs, the house of cards will come crashing down. people who have bought into the game have internalized this logic and will probably try to police you if you bring forgeries into any place that has a commercial interest in magic.
but i'm not interested in going to any of those places. i just want to play kitchen table, so is there any other reason not to use proxies? so far, the only negative i've found is that when every card is free, it's tempting to print some of the really strong expensive decks and these are actually less fun to play. a deck that performs well in the pro scene needs to be very consistent, which makes the experience of playing them kind of monotonous. still. i'm glad i didn't pay $500 to learn that lesson.
thus, arbitrarily, self-servingly, i'm okay with being a free-rider. there are 30,000 unique magic cards. if enough people join me and bring hasbro to financial demise, there will still be plenty of jpegs to go around. anarchic homebrew magic would still exist (even though i admit it would probably never be as well-designed and cohesive as the game is today) and if people really wanted, i'm sure mark rosewater and co. could start a new patreon-funded organization to design copyleft cards indefinitely.
would that work? i have no idea, but it'll never happen as long as magic exists in its current form, so i'll happily play the game for free until we maybe one day find out.
a classic thing that happens is someone says "i think the reason is blah" and then another person says "i think it's something else" and then a third person says "it's a bit of both" or "it's somewhere in between" or something like that.
it's such a reliable thing to be able to say and it takes so little effort, yet you can sound quite wise saying it. oh shit! it's both things?
i'm not trying to be derisive. it's just a fact of how conversation often works. it's like the rule of threes in comedy. it has to be three things because if you only have two things then there's no expectations to subvert.
euclid could have put it in the elements. look:
when three people at a party undertake the great pastime of trying to work out what the fuck is going on, one person will eventually propose their theory. someone else will then disagree in some way and suggest an alternative. then the Canny Third Person swoops in.
"I see the disagreement you two are having. Allow me: it's a bit of both."
"Oh, sick! Thanks, man!"1
no one is trying to be bullheaded in this process. it's just how these sorts of conversations have to be staked out. the mechanisms of the world are multifactorial, but no First Guy with a shred of compassion is going to enumerate their complex 10-factor weighted model of Blah to two strangers. they're going to say what they think the main factor is. then the Second Guy will dilligently remind them of the second main factor and how it's equally if not more important. then you step in and moderate and eventually we hone in on the amount of truth we have the patience to find.
or, you can instead make this meta observation about the mechanics of dialectics and really shut shit down.
party on.
i realised recently that there are at least two ways in which we ask the question "is x a y?"
it was when i asked "is Terminator a monster?" which is the sort of question i like to ask, to get into all the necessary-but-not-sufficients of dumb shit conceptual analysis. does a monster need to be biological, or more fundamentally, not understood? are all monsters morally permissible to kill?
there is a tweet that responded to Chess Is Not A Game by Deborah P. Vossen that i can't find. it said something to extent of "What the author fails to consider is that chess is, in fact, a game." because the other way we ask these sorts of questions is the Family Feud way. if you surveyed one hundred people with "Name a monster" - zero of them would say Terminator. they would say Dracula or Zombie or Frankenstein. and so in that sense, Terminator is not a monster. it is important to be considerate of your friends and realise this, when asking these sorts of questions.
but... y'know... if a new Terminator film came out that played with this idea, and used a lot of monster movie tropes, and it was a huge success, maybe twenty years from now a few people would think of the T-800 instead of Dracula.
a classic mistake people make is believing that the groups denote something as real as the things themselves. This is why I’m no fun when people ask shit like, “Is a hot dog a sandwich?” I know it’s a joke question. But what do you people want from me? Oh, let me contemplate the Platonic ideal of sandwich, which definitely exists, and then get back to you. I just don’t think that’s how language works. “Sandwich” is a word we deploy in specific situations to effect communication; it is a concept we have invented in order to do things. I don’t know if a hot dog is a sandwich, man; what I do know is that if you asked me for a sandwich and I brought you a hot dog, the likelihood that you’d be confused or irritated is higher than it would be if I brought you a PB&J. Does that mean a hot dog is only 70% of a sandwich? Angels on a goddamn pinhead, I tell you.
anyway. the reason i'm mentioning all this is because i just got back from a trip and an underappreciated way in which new places feel different is food.
food is, of course, one of the top answers on the board for the Family Feud version of the question "Why do different countries feel different?".
i mean supermarket food.
yes, restaurant food in different countries is different too, but i think this means less when travelling in The West as i just did. every restaurant in my city already tries to distinguish itself with its menu and branding such that when taken together, they all collapse into a single category of "good restaurants" that is no different from any other western city's. some cities have a restaurant that is uniquely good that mine doesn't have, but that doesn't move the needle overall as to why one city will feel different. i imagine this argument breaks down when visiting a place with vastly different restaurant norms such as japan, though i have not been there, so i don't know.
but as for a supermarket in a western city, i go to mine most days. i'm effectively doing spaced repitition on a thousand brands and products, the gestalt of which is a sensation of image-culture that isn't distributed anywhere else in the world. of course it's still all the same stuff: cereals, bread products, tomato sauces. what matters is that every label is different from the labels of the cereals, bread products, and tomato sauces that i've familiarised myself with over years of listing around the aisles of my local.
new town, new supermarket. nothing's in the right place. everything seems to have been misplaced.
it also contains all the food that i actually most often eat, that i will tend to eat less of when i'm not home.
so if you're back from a long trip and trying to get grounded again, go to the supermarket