00:06 We've already been exploring Swift's async/await APIs, mainly
focused on @MainActor and the bridging between a delegate receiving callbacks
from URLSession and the world of structured concurrency. But we haven't yet
looked at task groups, which make up the structured part of structured
concurrency.
00:54 With traditional tools like Grand Central Dispatch, we're used to
kicking off a job on some queue and hoping for the best. Working with structured
concurrency on the other hand — where a task will wait for all its child tasks
to finish before continuing — takes a very different mental model.
01:25 We're going to play with tasks by building a simple web crawler.
We'll crawl our own site, running locally to avoid accidentally overloading our
actual server, by fetching the first page's title and its outgoing links, and
then adding these links to a queue to be processed. We'll repeat this process
until we've crawled every page of our site.
We'll start by setting up the basic infrastructure today. Later, we'll look at
how to parallelize the crawler's work.
Crawler
02:01 We create a Crawler object, and we'll make it an
ObservableObject so we can use it with SwiftUI. The crawler exposes the pages
it fetched, along with their URLs, through its state property. Each Page
value holds a URL, a page title, and an array of outgoing links:
@MainActor
final class Crawler: ObservableObject {
@Published var state: [URL: Page] = [:]
func crawl(url: URL) async {
}
}
struct Page {
var url: URL
var title: String
var outgoingLinks: [URL]
}
02:47 The crawl method is async so that we can make it return when the
entire site is crawled.
Fetching a Page
03:04 But let's start with fetching a single page. In an extension of
URLSession, we write a helper method that loads data from the given URL and
returns a Page. We use XMLDocument to parse the data and to query the HTML
document for its title and links:
@MainActor
final class Crawler: ObservableObject {
@Published var state: [URL: Page] = [:]
func crawl(url: URL) async throws {
state[url] = try await URLSession.shared.page(from: url)
}
}
extension URLSession {
func page(from url: URL) async throws -> Page {
let (data, _) = try await data(from: url)
let doc = try XMLDocument(data: data, options: .documentTidyHTML)
let title = try doc.nodes(forXPath: "//title").first?.stringValue
}
}
05:16 By specifying "//title" as the path, we query all <title>
elements anywhere in the document. The stringValue of the first result should
give us the page title.
05:54 For the outgoing links, we query all <a> elements that have an
href attribute. We compactMap over the found nodes to mold them into the
URL values. For this, we filter out the nil values, and we check that a
valid URL can be created from the link. Also, a link may be relative, e.g.
/books, so we use the URL initializer that stores the page's URL as the base
URL:
extension URLSession {
func page(from url: URL) async throws -> Page {
let (data, _) = try await data(from: url)
let doc = try XMLDocument(data: data, options: .documentTidyHTML)
let title = try doc.nodes(forXPath: "//title").first?.stringValue
let links: [URL] = try doc.nodes(forXPath: "//a[@href]").compactMap { node in
guard let el = node as? XMLElement else { return nil }
guard let href = el.attribute(forName: "href")?.stringValue else { return nil }
return URL(string: href, relativeTo: url)
}
}
}
08:39 We now have all the ingredients needed to construct a Page:
extension URLSession {
func page(from url: URL) async throws -> Page {
let (data, _) = try await data(from: url)
let doc = try XMLDocument(data: data, options: .documentTidyHTML)
let title = try doc.nodes(forXPath: "//title").first?.stringValue
let links: [URL] = try doc.nodes(forXPath: "//a[@href]").compactMap { node in
guard let el = node as? XMLElement else { return nil }
guard let href = el.attribute(forName: "href")?.stringValue else { return nil }
return URL(string: href, relativeTo: url)
}
return Page(url: url, title: title ?? "", outgoingLinks: links)
}
}
Displaying Crawled Pages
09:07 In ContentView, we show a list of the crawled pages, sorted by
their URLs, and we call crawl in a task attached to the view:
struct ContentView: View {
@StateObject var crawler = Crawler()
var body: some View {
List {
ForEach(Array(crawler.state.keys.sorted(by: { $0.absoluteString < $1.absoluteString })), id: \.self) { url in
HStack {
Text(url.absoluteString)
Text(crawler.state[url]!.title)
}
}
}
.padding()
.frame(maxWidth: .infinity, maxHeight: .infinity)
.task {
do {
try await crawler.crawl(url: URL(string: "http://localhost:8000/")!)
} catch {
print(error)
}
}
}
}
12:27 We need to update our app's settings to allow outgoing
connections. Otherwise, every network request would result in a failure:
12:38 We also add the following compiler flags to the "Other Swift
Flags" field of the build settings to opt into compiler warnings about
concurrency:
-Xfrontend -warn-concurrency -Xfrontend -enable-actor-data-race-checks
13:10 This triggers a warning about ContentView being non-sendable. In
other words: it isn't safe to be captured by the task closure, which is what
we're implicitly doing by referencing the crawler property. We work around
this issue by adding the crawler object to the closure's capture list:
struct ContentView: View {
@StateObject var crawler = Crawler()
var body: some View {
List {
ForEach(Array(crawler.state.keys.sorted(by: { $0.absoluteString < $1.absoluteString })), id: \.self) { url in
HStack {
Text(url.absoluteString)
Text(crawler.state[url]!.title)
}
}
}
.padding()
.frame(maxWidth: .infinity, maxHeight: .infinity)
.task { [crawler] in
do {
try await crawler.crawl(url: URL(string: "http://localhost:8000/")!)
} catch {
print(error)
}
}
}
}
13:54 When we run the app, the first page is fetched. Next, we need to
take the local URLs from the page's outgoing links and add these to the crawling
queue. The processing of this queue is what we want to parallelize in the end,
but we'll crawl one page at a time at first.
Crawling Queue
14:44 We create a Set<URL> for the queue, and we start with the
requested URL as its only element. In a loop, we take the first URL from the
queue and load its page:
@MainActor
final class Crawler: ObservableObject {
@Published var state: [URL: Page] = [:]
func crawl(url: URL) async throws {
var queue: Set<URL> = [url]
while let job = queue.popFirst() {
state[job] = try await URLSession.shared.page(from: job)
}
}
}
16:11 Then, we filter the page's outgoing links so that we only have
"local" URLs — i.e. the ones that start with the URL we're crawling. We also
discard any URLs we've already seen. The remaining URLs are then added to the
queue:
@MainActor
final class Crawler: ObservableObject {
@Published var state: [URL: Page] = [:]
func crawl(url: URL) async throws {
let basePrefix = url.absoluteString
var queue: Set<URL> = [url]
while let job = queue.popFirst() {
let page = try await URLSession.shared.page(from: job)
let newURLs = page.outgoingLinks.filter { url in
url.absoluteString.hasPrefix(basePrefix) && state[url] == nil
}
queue.formUnion(newURLs)
state[job] = page
}
}
}
18:52 Now we're getting more results, but the list has a lot of
duplicates. The crawler also takes a long time to finish, because it may be
stuck in a loop.
19:11 Before fixing these issues, we add an overlay to show the number
of crawled pages, which will tell us if the crawler is still running:
struct ContentView: View {
@StateObject var crawler = Crawler()
var body: some View {
List {
}
.overlay(
Text("\(crawler.state.count) items")
.padding()
.background(Color.black.opacity(0.8))
)
}
}
Removing Duplicates
20:30 There are multiple reasons why we're getting duplicate pages, one
of which has to do with how we construct a URL from an outgoing link. It's
possible that the initializer of relative URLs constructs different values for
different base URLs. Let's say we're fetching the /about page and it has a
link to /books. The URL we construct there might not be equal to the link to
/books we find on /blog, even though they point to the same books page in
the end. We can avoid these duplicates by converting all URLs to absolute ones:
extension URLSession {
func page(from url: URL) async throws -> Page {
let (data, _) = try await data(from: url)
let doc = try XMLDocument(data: data, options: .documentTidyHTML)
let title = try doc.nodes(forXPath: "//title").first?.stringValue
let links: [URL] = try doc.nodes(forXPath: "//a[@href]").compactMap { node in
guard let el = node as? XMLElement else { return nil }
guard let href = el.attribute(forName: "href")?.stringValue else { return nil }
return URL(string: href, relativeTo: url)?.absoluteURL
}
return Page(url: url, title: title ?? "", outgoingLinks: links)
}
}
21:33 We're also seeing many URLs with fragments. We can remove these
by stripping any suffix that starts with a # character:
extension URLSession {
func page(from url: URL) async throws -> Page {
let (data, _) = try await data(from: url)
let doc = try XMLDocument(data: data, options: .documentTidyHTML)
let title = try doc.nodes(forXPath: "//title").first?.stringValue
let links: [URL] = try doc.nodes(forXPath: "//a[@href]").compactMap { node in
guard let el = node as? XMLElement else { return nil }
guard let href = el.attribute(forName: "href")?.stringValue else { return nil }
return URL(string: href, relativeTo: url)?.simplified
}
return Page(url: url, title: title ?? "", outgoingLinks: links)
}
}
extension URL {
var simplified: URL {
var result = absoluteString
if let i = result.lastIndex(of: "#") {
result = String(result[..<i])
}
return URL(string: result)!
}
}
23:01 For some pages, both a URL with a trailing slash and one without
are listed. So we remove the trailing slashes from all URLs:
extension URL {
var simplified: URL {
var result = absoluteString
if let i = result.lastIndex(of: "#") {
result = String(result[..<i])
}
if result.last == "/" {
result.removeLast()
}
return URL(string: result)!
}
}
Coming Up
24:28 The crawler now finds all pages once, extracting titles and
outgoing links. But the crawl method loads just one page at a time. In the
next episode, we want to parallelize this process by running multiple child
tasks at the same time. Since these tasks will operate on the same queue, we'll
have to use actors to synchronize access to the queue. We'll also need task
groups to wait for all child tasks to be done.
