0:00:24.8 Samuel: Yeah, so I have that happen to me all the time too, I'm like, "Why isn't my code working? Why isn't my code working. Oh, I didn't rebuild the app, it's the old thing, I fixed it." I wanna get one more joking 'cause I just thought of it. I got a haircut last week. Does that mean I'm supposed to be like going on a date this weekend? The numbers would say yes.
Yeah, so hi. We're gonna talk about Build Systems today. I have to apologize, the title was too good for me to pass up "from source code to executable: a day in the life of a Build System". This isn't going to be like a Hallmark movie, even though it feels from the title like it should be. So about me, I'm Samuel. You might know me. And my handle @SEGiddins from places like CocoaPods, I was a former core contributor bundler, which I maintained for a while, Bazel, I'm still a maintainer of the official Apple and Swift Bazel rules and Twitter where I tweet.
[laughter]
0:01:53.5 Samuel: So today, a quick run-down of what we're gonna talk about, what is a Build System? Why do we need Build Systems? How Build Systems work to describe builds, how they execute builds, and what's the future of Build Systems? So to start off, what is a Build System? Sort of deceptively simple question. Does anyone here want to take a guess as to what a Build System is? One sentence or less.
0:02:29.0 From the audience: A system to build things.
0:02:31.1 Samuel: A system to build things. Yeah, that's pretty close. So the way I'm gonna choose to describe it is a Build System, and because I have a sort of mathematics background to Build System is a function that takes in code and spits out something that we can execute.
0:02:53.8 Samuel: So did you know that you use a Build System, every day you work. Shocker, talk to your doctor and ask if Build Systems are right for you. Side effects may include kernel panics, Xcode crashes and a lot of swearing. So why do we need Build Systems? I'm gonna sort of quickly answer this question by example and trust that you'll be able to take the simple example I show and extrapolate it from your own experiences and believe that we do indeed need Build Systems. So here is our incredibly complex program, written in Objective-C, this is like a world-changing program. Took me 30 hours to write.
0:03:50.1 Samuel: And so let's say that we wanna build it and produce a command line program that we can run. So here's how we're gonna go about building our executable named executable from our source file. So it's Objective-C, so we are going to run clang, we're going to tell it the file that we're building, we're going to tell it that that language it's written in is Objective-C. We're going to tell it to output a file named executable, and we're going to tell it to enable modules.
0:04:26.6 Samuel: Okay, that slide had four different steps in it, and this was supposed to be the simple example. Even then it's not so simple. So I know you can't read this. I sat in the back row yesterday, I'm sorry. If it makes you feel any better, I couldn't read the slide either when it was sitting on my laptop, and that's kind of the point. This is, if you add -v to that command line (verbose) that I showed before, this shows you what clang is running under the hood, and it's running two dozen lines of code from our one-line clang execution earlier. And as an aside, yes, this does mean that clang itself is a Build System.
0:05:18.7 Samuel: Pretty meta. Pretty cool. Sure. So getting back to our question of why do we need Build Systems? Build Systems help manage that overwhelming then, and it truly was overwhelming, you saw that last slide, list of flags that you pass to build tools, figures out which tools turn which inputs into which outputs, and it coordinates that pipeline of build tool invocations so that everything is built in the right order every time you hit the build button in Xcode. So we know what a Build System is. How do we tell a Build System what to do?
0:06:02.3 Samuel: So for iOS developers, we use Xcode. We tell Xcode what to do, and it sometimes listens. These are going to be the most popular build description formats that we use for building iOS code today. Number one, obviously, is Xcode projects. You also have Swift Package Manager package.swift files and Podspec files. So, what do all of these build description formats have in common? Well, you describe one or more targets with them. Like say, you have an app, you have a framework, you have tests. You have tests, right? You list the files that are compiled into each target. You describe the dependencies between targets. Your app requires the framework. The tests require the app which requires the framework and so on, and you list the end products that the targets produce.
0:07:13.8 Samuel: So, in Xcode, this is pretty straightforward. Every target has a product, you have an app target it produces an application, you have a test target it produces a test bundle, and so on. Swift Package Manager actually does a good job of showing that products and targets are kind of different because in Swift package file, you can have multiple targets that don't relate one to one to products. But anyway, here... Let's go through some examples of what this looks like. We'll start with an example from the SwiftLint project. So, this is a makefile that can build test and package SwiftLint. Make is maybe the second simplest popular Build System. The first simplest, of course, is you write a shell script, and it's horrifying, and you never ever, ever, ever, ever, ever, ever wanna debug it. Makefiles are one step better than that. I'll accept an argument that they're also one step worse, but this is sort of the rest of SwiftLint's makefile.
0:08:32.2 Samuel: And I edited it out to try and fit some stuff onto the screen. Hopefully, you can read some of it. So, you can notice that make has targets. The target names are the things that go before the colon. So, targets here are like clean or clean Xcode, or build. Make also has a dependencies. So, these are the things that come after the colon. So, we can see that when you build that depends on clean and it depends on build x86-64 and build arm64. And it also has actions. So, for example, when we build arm64, we run swift build. SwiftLint also has a Package.swift file, so Package.swift is the Swift Package Manager's package description format. So, this Package.swift, which again, sort of edited for brevity describes how to build the SwiftLint executable, the SwiftLint framework and the SwiftLint framework tests.
0:09:46.1 Samuel: Notice that it's sort of much more semantic than the makefile. It describes what we want to build less so than how we want to build it. There is nothing here that calls swift build, or xcodebuild or swiftc. Instead, we say, "I have a package," this package has a name, can build for a couple of platforms. It has these products, it has this SwiftLint executable product, which depends on the SwiftLint target, the SwiftLint target has a dependency on these frameworks. And when we say that dependency, we literally just say, "Hey, here's a dependency on another swift package." It's really focused on explaining what the build is trying to do, rather than the imperative steps to perform the build. And finally, for completeness's sake, this is the Podspec file that's in the SwiftLint repo. This doesn't actually build SwiftLint, it only downloads the SwiftLint executable. So, you can then like run it outside of Xcode.
0:11:06.3 Samuel: But overall, it's pretty similar in being semantic to the Package.swift file. We say, "Here are the files, here's the targets we work for, here's a version and so on." And of course, I saved the best example of how we describe iOS builds for last. Who here has messed around with one of these views before and come away? Okay, and come away really frustrated?
[laughter]
0:11:43.6 Samuel: More hands come up when I add the frustrated part. Great. Yeah, so this is an Xcode project. And this is the Xcode project editor. And I'm sorry, I should have also taken a screenshot of the underlying file format for Xcode projects. Because if you think the GUI here is terrifying, the open step plist format that describes a .pbxproj file is even worse. As someone who's written a parser for those files, they're terrifying and I can't believe Xcode still uses them. So, okay, we've got all of these ways of describing our builds. We can write a makefile or write a Podspec or Package.swift file and list out our targets and their dependencies on their products.
0:12:50.3 Samuel: That, sure sounds like a lot of work. Again, show of hands, who's spent like more than an hour trying to edit a Podspec file or a Package.swift file to make it do what you want? Yeah. A lot of people, that's a lot of effort to go through for frustration. What's the point of doing that? And the point of course is that we can run a build without needing to list out every single step that you need to do in the correct order by hand. So a quick detour. There's this idea of dependency graphs and it's sort of the core concept that Build Systems deal with. So imagine that each one of the nodes in this graph are targets and arrows represent a target depending on another target. Maybe we've got R1 here is our application, and R2 is a framework, and R5 is another framework.
0:14:02.8 Samuel: And our application depends on a framework, which depends on another framework, but that framework is also dependent upon by a different framework. And are you getting confused yet? If you are, like this is the sort of thing that having Build Systems helps us deal with. They're really good at interpreting and manipulating and operating on these dependency graphs. Which can get complicated. They can get so complicated that when you have graphviz output in SVG of your dependency graph, it can crash safari and it can crash preview. Don't ask me how I found that out.
[laughter]
0:14:44.0 Samuel: So, okay, we have our Build Systems and we wanna execute build with them. Step one is we have to figure out what's going on in the build like that build description, we need to read it, we need to interpret it, and then we need to translate from that build description to a graph that sort of describes inputs, outputs, and actions. Inputs and outputs you can sort of think of as files. Actions are basically things you can run on the command line. As an example, let's say I have a file called input.txt, and I wanna make a file called output.txt, and the output should be the same as the input. Then my action would be like "cp input.txt output.txt." So we build up a huge graph of these actions. Once we have that graph, we know the thing that we're trying to build. Let's say we're trying to build our app. We have a graph. We know that the app depends on the framework. The framework requires us to build a swift module, to build a swift module. We have to build each swift source file in that module and then create a swift module from the source files.
0:16:08.9 Samuel: So we find the tasks that we need to operate that are unblocked, meaning all of their inputs exist. We've run the actions to generate their inputs. We build the unblocked tests, and then we sort of go to step two and keep repeating that. And as long as our sort of goal product, the thing that we're trying to build, the thing we've selected in Xcode and hit build on isn't built yet and we haven't hit any errors. So it sounds simple, right? And I guess I really like using this phrase and, when I talk I apologize. The point of course is every time I say it sounds simple, I'm gonna show how it's not simple. So translating from a build description to that like action graph is really hard. It turns out. Xcode used to have these things called XC spec files that would describe how to build targets. It would say, "Okay, well if you have this build setting, here's how we translate it into these build flags."
0:17:30.4 Samuel: Everything in header search paths has -I appended in front of it and so on and repeat that for every language that Xcode knows how to build. And every build setting that Xcode knows about. It starts to get really complicated. So, as an example, I can ask you what does it mean to build a framework? And you could say, "Well, Sam, that's very philosophical." But what it means is you have an executable and an executable goes in a certain place. It goes in, framework.framework/framework. A bad example, maybe I shouldn't have picked framework as the framework name. Headers go into certain place, food.framework/headers, and so on.
0:18:30.1 Samuel: You could ask, "What does it mean to have a dependency on a static library?" In Xcode, you just add, you can drag and drop your library in to a link target with libraries build phase. And what does that mean? Well, Xcode knows that it means you have to add a -l to the linker command to say, "Here's a library search path." And then you have to add, -library and then the name of the library, but then the name of the library without the dot a at the end to link it.
0:19:13.1 Samuel: What default flags does each file need to be compiled with? Well, as we saw, going through and compiling Objective-C, yes, you're on clang, but every time you're on clang and you want it to be Objective-C, you have to tell clang, "Hey, -X Objective-C." Xcode actually now has a new format that it generates called the PIF or the Project Interchange Format that the new Build System consumes. I guess, am I allowed to still call it the new Build System? 'Cause I guess the old one's gone by now, but to me, it'll always be the new one. And Xcode knows how to generate that PIF file from an Xcode project. The execution phase that builds go through also gives you a lot of room to get fancy. And when someone who works on Build Systems says fancy, that's usually a code word to mean faster, because at the end of the day, what most people really care about is... You know that XKCD comic, where people are standing on chairs and have swords, and they say, "Get back to work." And it's like, "My code's compiling?" We'd rather our developers be writing code than fighting with swords, and so we've invested a lot of time in making builds faster, so you spend less time saying, "My code's compiling. I can't do any work right now."
0:20:55.1 Samuel: So getting fancy here means doing things like executing different tasks in parallel. If you have multiple... If you're compiling multiple Objective-C files and they're in the same library, they don't have any dependencies between each other, we have these beefy, amazing Apple M1 machines and they can run what? 16 things in parallel, that means our code builds 16 times faster. We can get even fancier. We can execute our compilers on other people's machines, which is better than executing them on your machine because your machine then won't turn into a lap warmer, and it turns out there's a lot more of other people's machines in the world than there are of your machine. Actually, back in the day, Xcode used to sort of have this distributed build option and then they got rid of it. And I don't think anyone noticed. Another way you can get fancy here is with caching. You can say, "Well, I've compiled this file before, and I've compiled it with all these different inputs and with these flags, I know what the output is. When you tell me to run this command, instead of running a really slow compile, I can just grab what the result would have been from a cache." Another thing you can do is, I said translating build descriptions could be difficult, well, it can also be slow.
0:22:34.4 Samuel: You can interlace interpreting a build description with execution, so you can start executing things sooner in a build. You've ever seen the generating project build description progress bar in Xcode? And it just sort of seems to keep going and adding a new number. It's like you're zero out of one, you're at 10 out of a 100. This is the sort of thing that can take time, and if you have to do it at the start of your build, makes your build slower. So let's say that we're interested in getting fancy, I like getting fancy. So what do we have to look forward to? The obvious answer is Bazel. Bazel is Google's open source Build System. It has this tag line of, "Fast and correct, choose two." That's a play on the computer science joke of, what is it like? Fast, secure, correct, choose two. Well, the idea being, you can't have it all. Oh, with Bazel, you can have it all. So Bazel is a general purpose Build System. It's actually used to build a lot of really large iOS apps today. For example, the Square app builds with Bazel. I used to work on that. I started the project to build it with Bazel. Airbnb, Lyft, Uber, etcetera, all build their iOS apps now with Bazel, instead of using Xcode. So a quick plug, I gave a workshop on building iOS apps with Bazel at App Builders 2022.
0:24:34.7 Samuel: It goes over how you can translate a simple iOS app into Bazel. There's a GitHub repo that sort of walks through the different steps. If you're interested, it might be a good place to start and look and play around. So this is an example of what a Bazel build file looks like. It describes targets, Bazel as this notion of rules, so you can say, see here that we have a Swift library rule, and that describes how to build a Swift library, we have an iOS application rule that describes how to build our iOS app that has a dependency on the Swift library. This actually, if you could read this again, it's kind of small, I apologize, it would look pretty similar to what a package.swift file looks like. A lot of the same fields get used.
0:25:32.8 Samuel: So Bazel is, I guess I'm just doing some marketing for the Bazel project today. So Bazel is a pluggable Build System. What that means is you're not limited to a pre-defined list of things you can build, even if, let's say you are adventurous and you wanted to build an Android application with Xcode. I don't know why you'd wanna do that, but if you are interested in it, you can. There's really no great way to tell Xcode, here's how you build an Android application, here are all the steps you have to take. Bazel is also customizable, and that you can say like, actually, when you're building an iOS app, I want you to go through all these extra steps instead, every time you build an iOS app, make sure you do dead-code stripping or something like that.
0:26:27.4 Samuel: It's built for monorepos. That might be a positive. I don't know why I still haven't worked out my feelings with monorepos, that's probably something I should discuss with my therapist later. [laughter] It's Google scale, because it's built by Google, and it's inherently like a multi-language and multi-platform thing, so you can build all of your different applications, iOS, Android web, front-end, back-end with Bazel in one repo, in one build, you could build it all in one command if you wanted to, if you have the patience to sit for an hour or two or three, I guess. Another answer of sort of what's next, is this with package manager, it's gotten a lot more powerful for the past couple of years with support for things like plug-ins and... It's probably the future of what building iOS apps looks like, in fact, I was talking with someone over lunch, and I might be behind the times, it might sort of be the present of what things look like rather than the future. So... Yeah, all that being said, I guess what really comes next is, Simone comes up on stage and asks me some questions that you all have. So, thank you.
[applause]
0:28:04.8 Simone: Thank you, Samuel. So actually the question is what comes next. [chuckle] So thank you for the question. Is there still space for innovation in Build Systems and in which axis?
0:28:26.7 Samuel: Yeah. I think this is gonna be a space that there's always room for innovation. It would be like asking, Is there's still room for innovation in programming languages? I think the room for innovation is always going to be looking at what are the problems that we have, what are the pain points, and then building systems that are really purpose-built to solve them. For example, Bazel exists because Google had this pain point of, Well, we built ourselves this massive monorepo, how can we build source code in it, and there wasn't a tool that was built for monorepos and now there is. I guess we'll see what, sort of the next set of pain points is, and eventually we'll see tools that are purpose-built to address them.
0:29:17.7 Simone: Speaking about Bazel, do you recommend Bazel to any medium-sized project, or not and do you have like to be, have a guru-like set up to have that.
0:29:31.0 Samuel: Yeah, I'd say Bazel's a really powerful tool, but it's a tool that requires a lot of investment and attention and care, if you're at the point of taking someone on your team and saying your job is to solve build problems and CI problems, then Bazel's a reasonable thing to look at. If you just wanna say, spend a week on something and then never touch it again, Bazel's not your thing, it's something that requires constant investment.
0:30:11.4 Greg: So Build Systems is something that we use everyday, but not everybody is digging into it. Why did you want to dig into the Build System itself?
0:30:22.6 Samuel: I got Nerd's sniped. [laughter] I guess that's a pretty common story for people with a build engineering background. I was in university and I started contributing to CocoaPods and there was this open issue about basically, We need to rewrite our dependency resolve, and one Friday night, I stayed up late and tried to do that and I didn't succeed then it turned out it was like a multi-month project of full-time work, but yeah, it's like... It's this really cool intersection of building tools that work for people and solve their actual problems, so it's the intersection of that and interesting algorithms and computer science stuff.
0:31:17.3 Simone: The question from the audience, which is again, about Bazel... We think it's very, very interesting today, and many people have maybe experimented with that. Do you have any way to tackle the learning curve and some resources that you can share?
0:31:35.6 Samuel: Yeah, so I think there's a bunch of resources, none of them are super comprehensive, unfortunately. It's... It maybe a common theme of Google Open Source projects. There isn't a great like, getting started for Bazel, but you also have like an application that you wanna build with it versus the example app that's just hello world. But there are some tutorials I linked in my slides, there's the workshop I gave, and I probably should write a blog post that builds on top of that, who knows if I will, but there's also a great set of blog posts that dive into some of the particular pieces of how Bazel works and how to get it set up with Xcode. I know the lift folks put some out, and that was really how my starting point for experimenting with getting Xcode and Bazel working together was their blog posts.
0:32:41.9 Simone: And, speaking about something which is more familiar to us about SPM, what's the direction it's going to and what we anticipate in the next foreseeable future?
0:32:53.9 Samuel: Yeah. So it's hard for me to answer that given, I don't work at Apple, I don't work on SPM, I don't know what their sort of internal roadmap looks like, but if the past is any indication, the future is making SPM work for more people and more situations. When it first came out, there were a lot of people who were like, Hurray, I can stop using CocoaPods now, SPM exists, and the reality was that, that wasn't the case for four or five years, there were these set of features that a lot of people relied upon, and many of those have been implemented now. There's support for resources, pre-built binaries and so on, and with the new SPM plug-in support, there's also room to do a lot of the really cool stuff that using tools like Bazel gives you, like say, doing code generation based on Protobuf files or GraphQL or things like that. So I think the future is being able to bring more and more of the existing ecosystem and have it work in SPM and have it solve the... Those problems that are common, but less common than the ones that they've already solved.
0:34:24.8 Simone: So I think that's all, thank you very much. Thank you for being here
0:34:28.4 Greg: Thank you, Samuel.
0:34:28.6 Samuel: Thank you!