I got a gig to work on a bot that should work with artstation.com website. The requirement was that there must be some automated way to send messages to users of ArtStation which will be provided. So I used AppleScript to interface with Safari and JavaScript to bridge between AppleScript and the web page. The bot or rather the script will get a list of user's profile link from the local file which will be provided and then load them in Safari. For the initial launch, if the sender is not signed-in, the bot will sign in with the sender's credential loaded from the credentials file. Then the bot will invoke a couple of JavaScript bridging methods to simulate a click on the buttons, add text in the message field, trigger change event so that AngularJS recomputes the constraints and enables the submit button. Then it invokes submit button tap which will send the message to the user, and finally closes the tab and if all tasks were done, closes the browser.

After sending this, the requirement was revised. There won't be any list provided and the bot has to send different message based on category of the user and has to get all the users that the website has. So basically the bot is a crawler plus a messenger.

The app is a Cocoa app which now uses a web view to do the messaging part. This gives more fine-grained control over the web page loading events and such. The JavaScript communicates to the native code using WebKit message handler. There isn't much for the UI for the app as the main focus was on the functionality. I added some screens to view the details of the crawler, messenger and to configure messages and sender credential, which gets persisted to the database. I use web view because the website cannot detect if the request is coming from actual user or automated. And it does not require proxies because of IP filtering etc.

There is no developer API the website provides. Life would have been much easier otherwise. I did some debugging of the JavaScript the site loads and figured how their API service works. I wrote the crawler and frontier services in Objective-C. The crawler first gets the Anti-CSRF token so that the request can get through the Cloudflare security validation. Without the CSRF token, we will get a captcha. Then the bot calls the user v2 API which uses the same params the website uses to get the list of users which returns data in JSON format.

Now that we have the users list by category, we need to persist the data. I used FoundationDB with the Document Layer. FoundationDB powers iCloud, and the DB is distributed, fault-tolerant, scalable architecture is very promising. All of the DB setup went really well. Now I needed a MongoDB driver to talk the document layer. So I used the official MongoSwift library, but now Swift Package Manager refuses to work because it sensed the presence of Objective-C code. After wrestling with SPM and Xcode and MongoSwift, I just wrote the FoundationDBSevice, which is the persistence layer in Objective-C so that I can call directly into the MongoDB C Driver to work with the DB. If I had used only Swift, working the SPM would have been easy and could have used MongoSwift readily. Nevertheless, the bot now crawls the website saving users based on the category to the local FDB. To not DoS their website, I used the GKGaussianDistribution that comes with GameplayKit to generate a random number within a specified set of mean and standard deviation values, and uses this value along with current time to schedule the crawl. The same logic is used for the messenger as well, but with a different set of mean and SD values. The bot saves the state of each crawl so that the next time it starts, it can crawl (fetch) users from where it's left off. Users are messaged only once for a category. If a user belongs to multiple categories, the person will get different set of messages relevant to that category. The sender details can be added from the settings which are persisted in the DB except for the password, which is stored in the macOS keychain.

We can set a message template for each category and the bot will interpolate the string before sending the message to each user. Now the bot says Hi to ArtStation users. I did not feel right about getting the list of users from the website like this. But it's listed in the UI anyway. So it wasn't a violation per se.

The source code is available at GitHub.