tag:morgangallant.com,2013:/posts Morgan Gallant 2022-04-12T04:39:32Z Morgan Gallant tag:morgangallant.com,2013:Post/1817132 2022-02-22T17:00:00Z 2022-04-12T04:39:32Z Speed Limits, Programming, and Finding Direction

One of the most common sentiments that I get from people who've worked closely with me, especially recently, is that I'm a quick programmer. While I generally disagree with this, I do think it's true in a very select few cases, and I think it's worth unpacking a little further in a blog post. By doing so, I think there emerges an opportunity to write about some of my overarching philosophies and practices, both for writing software, and maybe even for life in general.

To start, I want to clarify that I don't think I'm an above average programmer, nor do I think I'm incredibly speedy overall. However, I do understand why some people may think that I am either of those two things.

First off, the majority of my work (especially these days) is within codebases which I've designed from the ground up. If you ask, I could probably tell you in excruciating detail how every bit of that code is structured, works with each other, and overall, is implemented. This sort of recall is only possible in smaller programs (e.g., sub-10k lines of code), and has already started (and will continue) to degrade over time.

Secondly, for the majority of the time while I'm actively programming, I'm not really thinking, moreso reciting. Thinking through implementation details in advance (perhaps on a walk, in the shower, etc.) can seriously improve both the quality and speed of a programmers work. This reminds me about a fun anecdote from the algorithmic trading world, where firms constantly fire their hot path (cancelling events near-microseconds before completion) to keep their CPU caches hot. It's a lot easier to respond to a real-time market event when you've had thousands (if not millions) of fake trial runs just moments before.

Bringing this back to the main point, I think the reason why I'm able to (at least appear) as a relatively quick programmer is because, especially recently, an absolute ridiculous percentage of my brain cycles (even when I'm not working) are about work. While I can't say that I think this is an overall very healthy practice, what I can say is that I'm incredibly happy doing what I do. I'm basically living my high-school self's dream; I get paid to work full-time on the stuff I'd be doing on weekends and evenings anyway.

I've been relatively lucky as I've have had more chances than most to explore different directions, and to really narrow down what I care about and what I want to work towards. In my case, it's largely human-computer symbiosis and the idea of enabling humans and machines to work together in new ways. First, and to be clear, this is a very privileged statement; very few people have the sort of freedom or flexibility to make this choice, and I'm very grateful to be one of those lucky few. Secondly, I think it's important to escape any sort of elitism here (especially prevalent in tech), different philosophies, principles, and directions should be celebrated and learned from, not looked down upon. You don't have to start a business-facing software company to be successful.

I'm a big believer in this concept of "there is no speed limit", which I believe was first introduced and/or publicized by a music instructor named Kimo Williams (source). This mindset of creating opportunities, rather than waiting or abiding by (artificial) speed limits, in my opinion, is a huge advantage for an individual (especially a programmer or a founder) to have. While others are blocked on frankly, stupid bullshit, a "don't wait" type of person will either continue to make forward progress in another area, or perhaps even rethink the problem and question the block to figure out ways to get around it and keep moving forward. This type of attitude compounds, especially as momentum and inertia come into play (don't let anything get in the way of this).

Frankly, I'm not really sure how to end this. I wrote this blog post originally to clarify my thoughts on this whole idea of a "quick programmer", and specifically, how to respond when someone calls me one. My initial attitude towards it was negative, largely stemming from too many pairing sessions and hackathons where my team/partner likely felt uncomfortable and unable to keep up with the speeds at which I was churning out code. Throughout writing this post, I think my overall attitude changed, I started to warm up to the idea, seeing it more as an advantage rather than something I need to work on or be ashamed of. If anything, I probably need to go faster.


]]>
Morgan Gallant
tag:morgangallant.com,2013:Post/1817134 2021-11-22T17:00:00Z 2022-04-10T15:47:09Z Beating the Crap out of the Stripe API (Respectfully)

Alternate Title: Fetching large volumes of data concurrently from a cursor-paginated API.

This past weekend was spent largely on one specific problem - how can we fetch huge volumes of data from the Stripe API as quickly as possible? To frame the problem a little better, it's probably worth giving some background. Our initial implementation was the simplest possible one (as it should be), which was roughly the following:

let subscriptions = [];
let more = true;
while (more) {
  let url = `https://api.stripe.com/v1/subscriptions?limit=100${
    subscriptions.length > 0 ? `&starting_after=${subscriptions[subscriptions.length - 1].id}` : ""
  }`;
  const response = await fetch(url, {
    method: "GET",
    headers: {
      Authorization: "Bearer " + operand.env("STRIPE_SECRET_KEY"),
    },
  });
  // omitted: error checking and other nonsense
  const json = await response.json();
  more = json.has_more;
  subscriptions = active.concat(json.data);
}

Essentially, we'd fetch the first 100 subscriptions, then if there were more, we'd fetch the next 100, until there are no more subscriptions left. This implementation worked fantastic in development, where we were using a test mode Stripe account with a total of, wait for it, 4 customers. I'm not sure what we were on that day, but we shipped this into production and onboarded one of the first external users onto the system. We learned of our mistake pretty quickly, as this specific user we onboarded ran a large business w/ ~60k customers per month (note: this is our conservative estimate based on the company's published stats). The above implementation is still technically correct for this volume of customers, yet we run into issues with the runtime speed. Assuming ~60k items, fetching 100 (the maximum allowed by Stripe) at a time taking ~300ms per request, we'd expect the fetch to take around 180 seconds (and it did, yet our HTTP requests timed out before we returned a result to the user).

Alright, 180 seconds is our baseline - how can we make it faster? We want this system to operate in largely realtime; users shouldn't have to wait for answers. Luckily, Stripe itself has pretty generous rate limits, 25 (read) requests per second in test mode and 100 in live mode. This means that there is ample room for us to speed this up considerably via concurrent requests.

When you hear the word "concurrency", what do you think of? Personally, I definitely don't think of JavaScript, rather, I think of Go. It's a language purpose built for this kind of stuff. I'm really sorry if you saw that JS code snippet earlier and got excited about some beautiful JS implementation, as there is something I failed to mention. That JS code is running inside a V8 runtime using the v8go package, and using the magic of polyfills, we can do our implementation of this in Go and call the function from JS. With this approach, our JS code becomes the following:

let subscriptions = stripe.fetch({
endpoint: "https://api.stripe.com/v1/subscriptions",
});

And yes, I am sorry, but that will be the last JavaScript code snippet in this entire blog article. We're operating in Go(land) now, meaning we get to use all the concurrency primitives of the language in addition to a bunch of useful open source packages.

The typical way of making these types of large-volume API requests concurrent is to use the (semi-standard) limit and offset parameters. This implementation works because you can spin up n workers to request blocks of 100n items at a time. Essentially, this means that even though Stripe limits you to fetching 100 items at a time per API request, you can make a bunch of API requests simultaneously to fetch a large range of items. Once you notice that you're not getting any more items back from Stripe, you're done and you can return the data to the caller.

The problem is that Stripe doesn't support the offset parameter in their API requests. They used to, but they switched over to using starting_after and ending_before parameters in 2014 (aka cursor-based pagination). This makes a lot of sense from a database perspective, as offsets are rather expensive (and doesn't scale well) whereas starting a scan from a particular (indexed) ID string is super cheap. For backwards compatibility reasons, although the parameter itself is depreciated, most (about 90%) of their main endpoints still support it (though this is largely undocumented). We weren't comfortable relying on a depreciated feature of their API long-term, so we had to figure out a way to fetch the data concurrently with their cursor-based pagination system.

This is the documentation for the starting_after parameter:

starting_after

A cursor for use in pagination. starting_after is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, ending with obj_foo, your subsequent call can include starting_after=obj_foo in order to fetch the next page of the list.

Wait a minute, we need to pass in the ID of the object? That's bad right? The goal here is to fetch data concurrently, yet how the heck can we fire off multiple requests at once if we need data from the previous request to create the next? We banged our head against the wall for a couple hours on this, went through most of the well-document stages of grief, and then in an act of desperation we started scouring the internet for ideas. Someone has definitely had this problem before right? Yes, yes they have. We found this really well written blog post by @brandur explaining some details about fetching data in parallel from cursor-based APIs. In his post, he suggested slicing the overall time range into pieces and fetching those pieces independently. He even mentioned the Stripe API specifically, the fact that they support a created parameter on all of their API requests:

created

A filter on the list based on the object created field. The value can be a string with an integer Unix timestamp, or it can be a dictionary with the following options:

created.gt
Return results where the created field is greater than this value.

created.gte
Return results where the created field is greater than or equal to this value.

created.lt
Return results where the created field is less than this value.

created.lte
Return results where the created field is less than or equal to this value.

This was exactly what we needed - we could split the overall search space into n pieces and fetch data from those pieces concurrently, using the starting_after parameter inside the pieces themselves to fetch all the data. In theory, this works great, yet a few concerns do arise (also partially mentioned in the blog post) which require some special attention:

Initial time ranges. We have no way of knowing a) how many items there are, or b) when the first item was created, so how do we choose our initial bounds for our requests? Distribution of items. In a perfect world, data is distributed uniformly throughout a given time range. This isn't necessarily the case for Stripe data. To explain this further, let's say you're fetching customer data from 2021. You split the time range into 12 equal segments, one per month, and fetch them all in parallel. It's possible that you had only a few new customers in most months, but in October, you had a blog post go viral and you got tens of thousands of new customers. In this case, the data itself is unevenly distributed, and the time slice for October will be stuck fetching all the data for that month in essentially the same manner as our initial implementation (single worker, 100 at a time). Put more formally, the runtime speed of the basic time slicing approach is limited by the biggest (& therefore slowest) time slice.

Since we had no idea of the distribution of the data, it was clear that we were missing something from the time range approach. Namely, we quickly realized that although we don't know the distribution initially, we do slowly learn about it as we do more and more requests.

Wait, pause, you said the word "learn" in a technical blog post? Machine learning? Training neural nets? Deploy on TPUs (plug)? No, no and no. I guess you could do that, but really, "learning" in this case means that we can reallocate workers from finished time slots onto those which require more attention.

Back to our fetching 2021 customer data example, where October has tens of thousands of customers and the other months have none. We initially take the naive approach, splitting the time range into 12 pieces and doing requests for all of them. When we get our results, we notice that 11 of the 12 months are empty, whereas the 12th month (October) returned a response with 100 items and a flag which tells us there are more requests. We now have 12 free workers, and the knowledge that we still have more items to fetch from October. At this point, we can take the time slice for October, slice it up once again, and repeat the same procedure (this time, each worker is processing slices of ~2.58 days). This gives us a way to continually do concurrent requests for time ranges that we know are important and contain information that we need to fetch.

This seems great, though there are some important considerations that should be mentioned, especially if you the reader are planning on implementing a version of this for yourself (it's fun!):

  • Choosing the subdivision parameters is important. For us, a value of 6 seemed to work well, though we haven't experimented with it too much. Essentially, this means that when we see a time range that has more than 100 items in it, we split that time range up into 6 pieces and queue up fetches for each of those time ranges.

  • After finishing the first request of a given time range denoting that there are more elements inside it to fetch, you don't need to slice up the whole time range, only the time range remaining after that initial fetch. For example, if we fetched the first 100 elements from October, rather then slicing up the entire month of October, we can look at the last fetched element for its created date and fetch from there to the end of the month, which means we aren't refetching the same data twice.

  • Depending on the distribution of data itself, it will be useful to have a parameter denoting the minimum size for a time range. Rather than further subdividing these ranges, we use the simple starting_after fetching technique. This lowers the overall number of requests required drastically because it prevents separate individual requests for tiny subranges, likely containing less than 100 items.

Here's some more code for your viewing pleasure (the variable ts is the timestamp of the last fetched element for a time range we're trying to slice):

// Since `gte` defines the bottom of the range, we selectively lower the
// top of the range using the `lt` parameter. If div is too small, we've
// hit the limits of our subdividing and we do bigger requests to ensure
// we're saturating stripeDatumsPerRequest (100).
div := (ts - elem.end) / stripeSubdivideRangeParam
if div <= stripeDivisionMinimumSeconds {
queue = append(queue, velocityStripeTimeRange{
start: elem.start,
end: elem.end,
after: id,
})
return nil
}
for i := 0; i < stripeSubdivideRangeParam; i++ {
tr := velocityStripeTimeRange{
start: ts - div*int64(i),
end: ts - div*int64(i+1),
}
// Little bit of nonsense to deal with gte/lt shenanigans.
if i == 0 {
tr.after = id // Important - don't want to fetch duplicate elements.
tr.start += 1
} else if i == stripeSubdivideRangeParam-1 {
tr.end -= 1
}
queue = append(queue, tr)
}

Another consideration is rate limiting - since you are firing a bunch of concurrent requests, you gotta make sure you aren't getting 429 status codes from the API. There's an excellent Go package for this exact thing.

// The actual rate limits for the Stripe API in test/live mode are
// 25/100 (read) requests per second. To be safe, we use only 80%.
var limiter \*rateLimiter
if strings.HasPrefix(skey, "sk*test*") {
limiter = &rateLimiter{
Limiter: rate.NewLimiter(rate.Limit(20), 1),
used: time.Now(),
}
} else if strings.HasPrefix(skey, "sk*live*") {
limiter = &rateLimiter{
Limiter: rate.NewLimiter(rate.Limit(80), 1),
used: time.Now(),
}
}

...

limiter.Wait(params.ctx)
results, err := doStripeRequest(params.ctx, url, params.skey)

...

It's also an interesting question to consider how to pick initial time ranges for requests. Our initial thought was to do it between unix time 0 and now, but this seems intuitively like a bad idea. The question then becomes, is there a date where we can be sure that no production Stripe data exists before it? Of course, it's the founding date of Stripe itself:

// The founding date of Stripe is 4/13/2009.
// Theoretically, no Stripe data should exist before this date.
var stripeFoundingUnix = time.Date(2009, 4, 13, 0, 0, 0, 0, time.UTC).Unix()

In test mode (with rate limits of 20 read requests per second) running on my laptop, we managed to fetch ~40k items from the Stripe API in 41s (which works out to an average ~9.75 requests per second, about 50% of the rate limit). We suspect this poor efficiency in our testing is likely due to the distribution of data, as it was created via a script and many of the Stripe data items were 'created' at the same time, making it challenging for our time-range fetching implementation.

This blog post will be updated with experimental data from live mode when we have been able to collect it. Our suspicion is that our efficiency (as a % of rate limit) will increase when working with perhaps more realistic data.

You can use a demo of this system (on a small amount of test data) here, and you can follow me on Twitter for updates. Thanks for reading!

P.S. If you're reading this and your name is Patrick Collison, please respond to my cold email 梁.


]]>
Morgan Gallant
tag:morgangallant.com,2013:Post/1817133 2021-03-12T17:00:00Z 2022-04-10T15:45:13Z Falling in Love With Silicon Valley

My flight to San Francisco touched down at around 10:30 PM on Jan 3rd, 2019 - this was my first time I had ever been to California. Air Canada had misplaced our planes luggage, so after waiting for a few hours at the airport, we finally arrived at our hotel in Millbrae at around 2am. My dad and I were both exhausted, it was a long flight and the difference in time zones made it feel like it was much later. We both woke up around 7am, and I remember vividly staring out the window, watching the sunrise over the water. I didn't know it at the time, but the warmth, energy, and beauty of that sunrise would foreshadow so much of my experience to come.

I was in California for an internship at Kuna, my first co-op term through the University of Waterloo. I had done internships before, but this one somehow felt different, felt real. Over the course of the next year, I would get the chance to experience some of what Silicon Valley has to offer, and meet people who would change everything for me.

This is the story of how I fell in love with Silicon Valley.

The thing that I noticed during my first week of work was the people - they are different from anyone I had ever met before. I didn't really know anyone, so I spent most of my time with my co-workers at Kuna getting to know them. Each and every one of them had this incredible focus, product obsession, optimism, and the confidence to try and make tomorrow better than today. It is this spirit that in my opinion drives so much of the success of Silicon Valley, this idea that through hard work, a small team of people can make a huge difference in the world as we know it.

Despite the overwhelming concentration of talent at that company, a few still managed to stand out above the rest. One in particular was my boss at the time, he had this aura around him, he shone (and continues to shine) like a bright light. He had gone through YC twice, the second time as the co-founder of Kuna. To be honest, one of the best parts of that internship was being able to learn how he thought through certain problems. He's very down to earth, honest, and most importantly, kind, and I think that's what made him successful as a founder. Generally speaking, he is the bright shiny thing that I point at when asked who I want to be like when I'm older.

Okay, so you met some cool people and you're a huge simp for your ex-boss, is that it?

Haha, no, definitely not. One of the coolest things about the valley is access, anyone can immerse themselves in Silicon Valley culture with one quick easy step, making a Twitter account. There is a tremendous community of builders, makers, and founders on "tech-twitter", and it was through this community that I met a bunch of people that I would now call my best friends. I remember that I had reached out to someone I had met on Twitter, and agreed to have dinner with them in San Francisco after work. I almost cancelled that meeting because I was too nervous, but I'm really glad I didn't. He had an incredible story, and again was someone that I wanted to be like when I was older. He ended up introducing me to Johnny, who eventually invited me to be one of the first ~50 or so members of Gen Z Mafia.

Gen Z Mafia, especially in the early days, was magical. It was this amazing group of people who I could relate to, and who were working together to build products that they thought needed to exist. I think the thing that surprised me the most was how ambitious some of the projects were. There was at least two banks being started at any given time, and countless other projects that would be laughable at best, even if an industry veteran with 30 years of experience were working on them. Yet there was these kids, not so different from me, actually doing this stuff successfully. I think so much of this "movement" was framed really well by the now-famed essay by Marc Andreessen, entitled IT'S TIME TO BUILD. The key idea behind the essay is that it's time for everyone to step up and fix the big problems that we as a society face, and I think a lot of people from the gen-z community, including myself, really resonated with that message. Although the server itself has largely died out, perhaps because of some core problems discussed here, I think the energy and movement that was started by that server will live on in almost every member, especially those present for the early days.

Many smaller groups have come out of that server, and I'm part of one of them - we don't like to talk about this too much, but this group is one of the most tight-knit, supportive, and generally driven groups I've ever seen. Out of the 31 people that are on the server, I think there are like 9 different startups being built who have collectively raised millions of dollars and are working on some really hard problems. We invest in each-other, with our time, money, and experience, and I expect this to pay off exponentially in the next few years. The plan is to travel the world together, and eventually get a house in San Francisco for the group. Being around people like this has made me into I believe a much better person, and as Johnny pointed out a few weeks ago, we've gradually become less and less active on the server because we've naturally become more and more busy. To note, this isn't a bad thing, we've all been pursuing more and more ambitious goals, and if anything, it has made us tighter-knit than ever.

That being said, I do think it is worth mentioning that there are problems with Silicon Valley. There is absolutely way too much money floating around. Sure, this is a great thing since the money will be used to fund the next-generation of ambitious companies, but I'm largely concerned about some of the patterns that have emerged as a result of this access to capital. For instance, so many companies these days, especially in Silicon Valley, tend to have hugely unsustainable business models, and rely on massive rounds to stay afloat. This makes sense for certain, perhaps more capital intensive or hard-tech companies, but it doesn't make sense for a small software startup, aka the canonical Silicon Valley company. It's hard to say no, especially to millions of dollars, but I think the best companies these days are the ones who are able to do this, and figure out how to make their product great without first raising an enormous round.

Another gripe that I have with the valley are the so-called imposters, the individuals talking about building without actually really building anything. These are the people who spend more on company merchandise than research & development, and hire the best frontend engineers to make a beautiful website because their product isn't good enough to stand on its own. I don't mean to put down anyone specifically here, I think overwhelmingly the people in the valley are generally building great things, but there are bad apples, and I very much hope that Silicon Valley as a whole isn't judged because of the actions of these few - they are the minority.

To conclude, the spirit and the energy of Silicon Valley is intoxicating, and I'm extremely grateful to be a part of this fabulous community of builders. I'm really excited to move to California, to not only be part of the gen-z lead movement building the next-generation of technology, but to finally be home.


]]>
Morgan Gallant
tag:morgangallant.com,2013:Post/1817135 2020-09-05T16:00:00Z 2022-04-10T15:46:43Z Back to School?

September 4th, 2020 — thousands of university students across Canada move into their new home for the next 4-8 months, yet campuses still seem empty, as if no-one moved in at all. Obviously there are still a few small groups of people wandering the streets in relative isolation, and of course the mad stampede for groceries from Costco is still in full effect, but something feels… different. 14 hours in, and university in 2020 feels like a puzzle that you’re trying really hard to solve, yet you only have about a quarter of the pieces, and some leftover pieces from the last puzzle you did. Sure, you can definitely make something out of it, but the end result will probably be misshapen and incomplete.

Fundamentally, I think the value of a university experience is a weighted sum between the relationships you develop while you’re there and the difference between your ability to learn new ideas from when you start to when you finish. I don’t think that university has anything to do with actually learning new skills, unless you consider alcoholism a skill. If you actually want to learn tangible stuff, either pickup a book (yes, the paper things — they still exist) or enrol in a college program / bootcamp. COVID-19 has affected both terms in the weighted sum that governs a university experience, and the rest of this thing (essay? blog? rant? not sure to be honest) will go into this in much more detail.

First up, the easy term — your ability to learn new ideas. University has traditionally been really effective at teaching students how to temporarily appear intelligent in a particular area. It doesn’t seem that the ongoing pandemic will affect this dramatically, since the migration of the academic experience to online seems to have gone relatively well. Students will still have to cram for exams, still have to assume that a textbook is the ground truth on a subject, and still have to memorize a lot of dumb stuff. All in all, probably a delta between -5/-10%, mainly due to the lack of in-person class and lab time.

Okay, that was easy. Now for the hard term — the relationships you develop. I’d argue that the delta here is between -70/-80%, and even higher for particular (rather unlucky) groups of students. Certainly some students (those in upper years) will already have some “puzzle pieces” to work with, mainly in the form of pre-existing relationships and avenues to meet people, whereas a first-year student has almost no defined pathway to acquire these invaluable “assets”. The same can be written using more technical language: first-year university students in 2020 are really fucked. Let’s unpack this a little further.

An upper-year student going to university in 2020 has the pre-established channels in place for near constant communications with their friends. This network also has the effect of creating new relationships between peers via mutual friends and introductions. Obviously, the actual effectiveness of online interactions is nothing compared to that of in-person meetups, dates, and group activities, so upper-year students will struggle (yet probably manage) to at least maintain pre-existing relationships and create a few new ones. In my circles, we’ve mainly used Discord as a way to stay in touch, whether it’s via text message or voice/video call. So far, this has worked pretty well. I’ll probably do a post later on some more specifics here, as there are some interesting dynamics which emerge in these type of server communities.

Unlike the upper-year student, a first-year most likely doesn’t know many people at the university, and won’t have any of these pre-existing communication channels. It will be the responsibility of a) the university and b) the student to ensure that these pathways are built. Normally, an orientation week takes care of this pretty well. The university basically shoves together a bunch of sweaty freshman into a big pit (complete with fun sponsored and not sponsored activities) and lets the new students figure it out among themselves. Naturally, small groups emerge (usually in similar programs or fields) which end up being study groups and most likely lifelong friends. This “ball-pit” experience isn’t possible over video/audio/text — are you going to just shove a bunch of kids in a Zoom call and see what happens? Of course not, that would be an absolute shit show. Rather, both universities and students will have to create opportunities for one-on-one communication possibly followed by a referral to join an existing social circle or meet someone new. As a side note, there is definitely a startup or two waiting to be built in this space…

If you’re a first-year reading this, you’re probably terrified, as you should be. Your university will most likely try really hard (probably too hard) to help you make friends, and you need to work hard at it too. It’s not okay to sit in your room alone talking to no one for all of first-year, you’ll get really messed up (I’ve seen it numerous times). Instead, get out there (virtually, and maybe physically) and actually talk to people and start to form your network. This network will be the most valuable asset you get out of university, so nurture it and don’t take it for granted.

The alternative for first-year students is to defer until 2021, which I honestly think is a great idea. Not only does this get you an extra year to figure out what you are actually passionate about (which could help explain why you’re going to university in the first place), but this essentially guarantees strong relationships with peers during first-year with little effort on your part (the university will probably do the heavy lifting here).


]]>
Morgan Gallant
tag:morgangallant.com,2013:Post/1817129 2020-08-05T16:00:00Z 2022-04-10T15:43:54Z Ethical Usage of Natural Language Tools

Anyone who has gotten beta access to OpenAI’s API, or has used any of the fantastic tools already created with GPT-3, knows that we’ve hit a major breakthrough in natural language understanding and generation. I would be remiss not to mention the famous Stan Lee quote saying that “with great power comes great responsibility” — now that we have powerful natural language tools, we need to make sure that they are used in ways that create a positive user experience and benefit society as a whole. The pressing question at the moment is where do we draw the so-called “line in the sand”, better put, what criterion do we use to judge whether or not an application is an ethical application of a natural language tool? Generally speaking, these types of ethical dilemmas are really difficult to solve (would you kill one person to save five?), so if you’re reading this, definitely don’t expect a concrete answer as to whether or not an arbitrary natural language tool is ethical or not. Rather, this essay will take a principled approach when thinking about these issues. It’s also worthwhile to mention that I think I’m uniquely qualified to write about this issue, given that I’ve built and released apps that have tested these ethical boundaries.

I got access to the GPT-3 API on July 13th, 2020 and released my first app using it by July 17th - a “complaint department” which responds to user complaints in the personality of Sergeant Gunnery Hartman from the movie Full Metal Jacket. I was (and still am) fascinated by the idea that it’s possible for GPT-3 to take on different personalities and emulate the behaviour of others, even fictional characters. Throughout my childhood, my father made sure that I had a great education of 80s/90s war movies, and one of my favorite characters from those movies is the drill sergeant from Full Metal Jacket, known for his clever, yet not so politically correct one-liners. I released this application with good intention, to give people a laugh and maybe even motivate them in some dark and twisted way to achieve their goals. However, at the time, I was certainly ignorant to a lot of the potentially negative impacts a tool like this can have. Even though the site had multiple warnings about how the generated content could be vulgar and offensive, even I had trouble not taking some of the responses to heart.

The site launched at 7:08PM on a friday, and by 7:39PM I had received a message from the Greg Brockman, CTO of OpenAI, asking for me to take it down temporarily and schedule a meeting with the team. In the time that it was online (about 40 minutes), the site got ~2.6k requests and I had received over 50 screenshots from people sharing notable responses. After a quick discussion with Greg and the team, it was decided that the site should remain offline for the foreseeable future, an action which I wholeheartedly agreed with. This was a great example of something that shouldn’t have been built with GPT-3, and as Greg put it, a developer should be ultimately responsible and ready to condone any outputs from their tools.

I had a video call with the OpenAI team the next week to talk about something new I was working on, and was extremely impressed at how the team is ensuring that GPT-3 is used ethically. Of course, they’re still working on developing their internal criterion for approving GPT-3 applications, but in general they’re taking a proactive stance on making sure that the technology is being used for good. After this meeting, I got approval to launch another tool which this time, hopefully did some good.

I launched Regex is Hard on July 31st, got a lot of positive feedback and got ~11.7k requests in the first 36 hours of launch. As the title suggests, Regex isn’t very fun to work with since it is a language designed to be parsed by computers. It was surprisingly easy (just 2 examples) to get GPT-3 to generate fairly accurate regular expressions from plain english, and can even (sometimes) go the other way. I think this is an example of a universally good use case for natural language tools, creating useful tools for others that help with a mundane or repetitive task.

I’m a big believer that every decision should be made through the lens of a set of strong personal moral principles that ultimately define who you are. Personally, I tend to take a utilitarian approach to solving most problems, except when doing so would be unethical. For a concrete example, I don’t believe that should’ve released the complaint department tool I wrote. Although it was intended to be comedic, it was hard to ignore just how much negativity was output from the tool, and how that negativity could harm the mental health of my users. It’s one thing to write a potentially harmful comment or joke, it’s another thing to tailor that towards a specific user using something as powerful as GPT-3.

Natural language applications are much more intimate with their users than other applications, because typically the user is able to have a conversation with the tool rather than just pressing buttons on a user interface. A great example of this which I do believe to be an excellent use of natural language technology is Inwords, a platform providing affordable, automated therapy sessions to “normalize and provide access to mental wellness” for its users. Technologies like Inwords work to use the power of natural language for good, and to develop close, meaningful relationships with users.

As I work to build new and more powerful natural language tools, I developed a set of four questions that I ask myself every time I’m about to release something new:

does the tool abuse any potentially intimate connections with its users? does the tool provide a universal benefit to each and every user in some way? if you, the developer, were asked to manually serve user requests without using any automated natural language tool, would you be comfortable doing so? can the tool be misused to unintentionally have a negative impact on users? If you can’t stand behind your answers to any of these four questions, it may be worth considering whether or not the tool should be built/released in the first place. I’m especially critical of the fourth question (potential for misuse) due to the relatively young age of powerful natural language tools — it seems we don’t have a complete understanding of the failure modes of these large models and the impact that it could have on users.

With the release of GPT-3, I’m fully expecting the next 1-2 years to be full of exciting new applications leveraging natural language to create powerful features. I’m looking forward to the day when I can write code by just describing my intention, or ask Elon Musk to teach me about rockets. I’m also really excited to be building and eventually releasing some tools of my own in the virtual assistant space. As we (the next generation of makers) set out to build these tools, it’s important for us to set a strong precedent of both building universally great tools with this technology, and being proactive in ensuring that the apps we build aren’t misused.


]]>
Morgan Gallant
tag:morgangallant.com,2013:Post/1817128 2020-06-07T16:00:00Z 2022-04-10T15:43:02Z The Future of Computer Networks

There is no doubt that computer networks, the internet (heard of it?), has changed the way that we live our lives. It started out very simply as a military experiment, which quickly began to see use within communities of researchers at different universities. As the internet grew, more and more people had access to this massively expanding repository of the world’s information. It's obvious that the initial plan for the internet wasn't to support this many connected machines (don't get me started on IPv4...), nevertheless, the internet itself is generally considered a good thing.

The problem with the seemingly never-ending expansion of the internet is somewhat paradoxical: a goal of the internet is to house the worlds information, yet a lot of that information shouldn't be on the internet. Consider a typical software company, they have a database server along with a few application servers which serve requests for their customers. Application servers and database servers are networked together, each of which are (typically) assigned a public IPv4/IPv6 address and put behind a firewall to prevent unauthorized connections. Have you figured out the issue yet? We put machines (possibly containing extremely sensitive data) on the public network, then try and patch up the holes with virtual duct tape. Then for some reason, we're surprised when vulnerabilities are found leading to the personal information of 147.9 million people being exposed (cough Equifax cough).

The solution to this problem is simple: some (if not most) machines shouldn't be accessible via the public internet. This really isn't a breakthrough as cloud computing providers have been pushing for this for a while now with VPC systems, and VPN technologies. The concept is relatively simple - small, self-contained networks which require authentication to join, meaning once you're inside, communications and data are private by default. In essence, what I'm trying to get at is this idea of a personal (or corporate) internet. Inside this personal internet, you're the boss, and you can control who joins this network, and who can talk to who within it.

At that, we reach the point of this (rant). Enter Tailscale - a startup I'm rather fond of, and who I have been a power user of for a while now. Tailscale lets you easily make your own personal internet, for free in most cases. It's a weird paradigm shift, since inside this internet, you must actively work to make your applications insecure rather than the other way around. All traffic is encrypted and uses P2P as much as possible (DERP servers are used when P2P connections aren't possible - essentially an encrypted packet relay, nothing special). For a fun anecdote that isn't condoned by my employer, I setup my work machine at the office to my personal Tailscale network, and was able to tunnel directly into the machine from my house, without being connected to any of the corporate VPN nodes (presumably through many firewalls / NATs). Pro tip - using a P2P connection for RDP sessions is much preferred over a VPN-proxy onto the corporate network, just don't tell IT.

On the topic of my employer (who makes some epyc gear), it's worth mentioning that adopting Tailscale would be a great move for the company. Right now, we have 17 VPN servers worldwide which allow us to connect to the corporate network. If we're not in the office, any connection we make to another server inside this network has to be through one of these VPN servers, which certainly introduces some difficulty given that most of our 11,400 employees are working at home due to COVID-19. Obviously, productivity is impacted by the load on these servers, which leads to RDP sessions being quite sluggish when connecting to our work machines. A Tailscale network would alleviate most (or all) of our networking problems, since any employee given access to the network would be able to make a direct P2P connection (for the most part) to any other machines on the network, if allowed to do so by the company set access policies. In addition to this, the company would also have full observability into the access logs of different machines, which could help protect sensitive information on the network. You could also say that it would be worth ryzen to the occasion (I'm sorry for that one).

Before I started at my current employer, I did two internships at a YC-backed home security company, which primarily made smart home cameras, video doorbells, etc. In this role, I primarily focused on live video streaming (bidirectional streaming between home IOT device and users smartphone), and let me tell you this: with a few new features (notably the ability to pre-authorize clients to connect to a network), Tailscale would've been a game changer. Latency was important (ex. it's important that users are able to have real-time conversations with people who have rung their doorbell), and the ability to easily make secure P2P connections between our servers and our cameras would've made my job 100x easier. Forget any NAT traversal systems, hole punching, or any other dirty networking tactic, it would all have gone away with Tailscale.

So yeah, with my experience at both an IOT startup and a major corporation with severe networking issues, I think I'm uniquely qualified to rant about how great Tailscale is. It's a surprisingly simple idea, let people create mini sub-internets that they control, and Tailscale has done an amazing job making this a reality. I couldn't imagine not using Tailscale for my daily work, whether it is SSHing into my home servers from anywhere or exploiting the fact that I own the network to do cool distributed systems work.

If you want to get started with automatic peer discovery with Tailscale, I've open sourced a little bit of code to help out with this! You're welcome.


]]>
Morgan Gallant