As it happens, my company uses TFS that is not git friendly yet. I quickly found git-tf to bridge the two repositories, but the job of manually syncing the local and remote repositories all the time and all the manual deployments were taking up too much of my development time.
Another thing that was making the deployments complex was the amount of changes I had to make in the configuration and the code when migrating from one environment to another.
So I quickly started working on a solution to automate my tasks as much as possible. Not because I'm lazy, but mainly because if we write the proper automation tasks, we avoid mistakes.
Here are my challenges:
1. Push notifications
Our apps use push notification and we use Urban Airship as our push notification server. As is well known, UA and APNS (Apple Push Notification Server) have two environments: Debug and Production. UA provides a configuration file where you can put the keys for both of these environments and at run time, it configures itself accordingly. The problem is that at my company, we have to handle 4 environments: Development, QA, User Acceptance Test and Production. All the pre-production environments have to use APNS debug, since they are not production, so I had to handle 3 different debug configurations for UA config file.
2. TFS integration
All the "other" code in the company is developed in .NET world. They get along quite well with TFS, SQL server, Visual Studio, Windows, but when it comes to git and Macs, the support is questionable. The company didn't like the idea of having a separate git server for mobile developments. They prefer the code to be imported into TFS and from there they can merge/branch according to the deployment environment.
As they have a lot invested in and knowledge of the TFS infrastructure, they want to benefit from it as much as possible.
3. Lack of Macs
Due to the cost of Macs, the company prefers to have mostly Windows workstations and laptops handed out to developers. We have very few Macs and usually they are handed out to executives that run a Windows VM on it all the time. They conceded that developers of iOS code need a Mac, but it was hard for them to accept that all dev ops would need a Mac too in order to build and deploy code to TestFlight and App Store. Having a Mac there is a sign of status that they were just not ready to spread to all levels of development and IT tasks.
4. Back-end integration
Our apps talk to Salesforce APIs on the back-end and again, each environment requires different endpoints and sometimes slightly different state machine to entertain the SOAuth dialog and certain customizations in the login process, like allowing the user to reset password from the login dialog ("forgot password link") and the always connected app.
5. Corporate network
My company is in the financial sector, so security is very strict. The network has all types of firewalls, traffic filters and restrictions they put in place for all users. The users are defined in Active Domain and the network is very Mac hostile. We have to change our passwords very often and every time it takes place, it is a nightmare on the Mac side. I have to either clear all sorts of cookies, certificates and passwords stored on the keychain and start over, entering the new password again a thousand times, or I have to painstakingly go through my key chain and update all the stale passwords. If you add to that the bug on Mavericks when your Mac is connected to redundant gateways, which disconnects every minute or so, remote tasks are very hard to accomplish on Macs here.
6. Maintaining the version string consistent across deployments
Uploads to the App Store require that the version strings (long and short) increase from one release to another. We also need to be able to identify uniquely each bundle uploaded to TestFlight or installed on a device. We need to look at the app and quickly identify which distribution is installed and which build generated that distribution.
Solutions:
Just trying to make my life easier, I started looking into ways to simplify my tasks with automation.
1. Push Notification
For the push notification, I wrote script steps into the build action on the scheme that updates the UA config file before the archive.
Here is an example:
# Archive pre-actions# set UA config to development/usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"if [ "$CONFIGURATION" == "Ad-Hoc" ]; then/usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber" "${PROJECT_DIR}/${INFOPLIST_FILE}"/usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_KEY" "${PROJECT_DIR}/AirshipConfig.plist" /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_SECRET" "${PROJECT_DIR}/AirshipConfig.plist" /usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"fiif [ "$CONFIGURATION" == "Production" ]; thenecho "actions before archiving for Production upload" >> ~/Public/${CONFIGURATION}MobilePartnerarchive.txt# change UA config/usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_KEY" "${PROJECT_DIR}/AirshipConfig.plist" /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_SECRET" "${PROJECT_DIR}/AirshipConfig.plist" /usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"fi
In this example, both ad-hoc and production of an enterprise program app use UA/APNS production, but they have different app bundle ids, which means they need to point to different topics in UA/APNS server,. For that reason, I need different keys/secrets for each distribution. In this example, we can populate the keys and secrets right before the archive, maybe using a script that captures them from the keychain on the build machine using the dev ops build user. The developer will never have access to these keys/secrets on her keychain.
2. TFS integration
For TFS integration I installed git-tf on the Mac. It is tricky to configure all the required java stuff and the symlinks to make it easy to update the versions in the future. You have to clone the TFS repository on the command prompt and if you don't configure the clone afterwards to store your credentials on the git configuration file, you will have to provide them every time you execute any git-tf command. You also have to create your .gitignore, to avoid grabbing all the Xcode temporary files into your TFS repository.
I couldn't find a way to call these commands from within Xcode when I'm in the source control menu. I think Xcode should provide a way to customize the actions there, similar to the schemes, providing hooks for pre- and post- action. This would have made my life a lot easier. You won't believe how many iOS and Windows developers are not familiar with command prompt in the terminal window.
The other bothersome feature was the fact that configuring git-tf to store the credentials would store the password unencrypted, and this configuration had to be changed manually every time we had to change our passwords in AD.
I started by writing scripts that would store the password on the keychain when cloning the repository and then would read the key chain to provide the password when checking in code to TFS or pulling in updates. They were still scripts that would run on the Terminal window and the developer had to remember to checkin code to TFS when she was ready to merge to other branches.
This also didn't solve the problem of automatically changing the password when the user had to change the password in Active directory. I'm still thinking about how to solve this problem of the password update triggered by AD.
The next step was to make the checkin to happen from within Xcode, without the need to run any scripts in the terminal window.
I usually checkin the code when I'm ready to deploy a binary to TestFlight, which means that I had generated an archive. As it happens, sometimes I would upload a binary to TestFlight and I would forget to checkin code to TFS, which would make the TestFlight code out of sync with TFS. I then decided to create a post archive script to checkin to TFS every time I generate an archive.
Unfortunately, git-tf does not return an error if the connection with TFS fails due to invalid credentials. In fact, if we keep providing an invalid password, it keeps asking for the password again 4 times before it ends without returning any error to the calling script. Which means that it can even lock the account depending on the policies of your company.
Since I didn't resolve the changing password problem, I decided to add a script that would always prompt the user for a password. At least the developer doesn't need to go to the terminal to run a script to checkin. Because I don't know how to show a prompt in Xcode without using AppleScript, that's the language I had to use to accomplish that.
Here is an example:
set automatedBuild to do shell script "echo ${AUTOMATED_BUILD:='NO'}"if (automatedBuild ≠ "YES") thenset archiveComment to "Xcode generated"set dialogResult to display dialog "Enter TFS password:" default answer "" hidden answer yes buttons {"OK"}set userName to text returned of dialogResultdo shell script "/bin/bash --login -c \"cd $SRCROOT/.. ; echo "& userName & " | git-tf checkin \" 2>&1 "display dialog "Code saved to TFS..." buttons {"OK"}end ifset archiveDir to do shell script "ls -dt1 $HOME/Library/Developer/Xcode/Archives/*/*.xcarchive |head -n1"trydo shell script "/usr/libexec/PlistBuddy -c \"Add :Comment string " & quoted form of archiveComment & "\" " & quoted form of archiveDir & "/Info.plist"on errorend trydo shell script "/usr/libexec/PlistBuddy -c \"Set :Comment " & quoted form of archiveComment & "\" " & quoted form of archiveDir & "/Info.plist"do shell script "/usr/libexec/PlistBuddy -c 'Save' " & quoted form of archiveDir & "/Info.plist"do shell script "/usr/libexec/PlistBuddy -c \"Set :Name 'QAbuild'\" " & quoted form of archiveDir & "/Info.plist"do shell script "/usr/libexec/PlistBuddy -c 'Save' " & quoted form of archiveDir & "/Info.plist"
This script assumes that the user has configured git-tf with her username, but it is not hard to change it to get the user name from the user running the archive in Xcode.
This code also changes the comment and the name of the archive file, so we can see on Xcode Organizer that this build was generated by Xcode and not by our automated build script.
I have the test of the Automated Build flag because the same scheme is used in our automated build solution that I'll explain later.
3. Lack of Macs
As I explained, Macs are hard to get at my company. But the fact that all the code I was working with had to be built on a Mac and that I was the only person with access to it, convinced my boss to invest in a Mac Mini server to be our build server. I had now to make it possible to remotely trigger the build and deployment on that machine.My first try was using Bots. It took me awhile to figure out how to use them. I demo'ed to my boss and he was very excited, because the dev ops could access the bots via web from their Windows boxes.
The major road block with Bots is that it requires git in order to work. I also didn't find the Xcode documentation very enticing and it lacks more meaningful examples. I did entertain the idea of scripts that would pull the code from TFS on the a git server on the build machine, but I haven't convinced dev ops of the feasibility of this solution yet. They were already developing workflows in TFS that queue the builds of all the .NET stuff and they opted for going that way.
I wrote a script that receives as arguments the environment of the build (DEV, QA, UAT, PROD), the scheme (I have some projects with more than one target) and it sets the identities, provisioning profiles, etc for xcodebuild to generate the archive directly from the command line. It also generates the .ipa file and uploads it to TestFlight if the build is for a non-production environment. The idea is to extend this build script to upload the binary to the Appstore for production environments of our public apps, but we are not quite there yet. This will remove the need for the dev ops to access a Mac except running remote tasks from Windows or accessing web servers.
Here is the example of the build script:
while getopts t:e: optiondocase "${option}" ine) CONFIG=${OPTARG};;t) TARGET=${OPTARG};;?|*) printf "Usage: %s: [-t value] comments \n" $0exit 2;;esacdoneif [ $OPTIND -gt 1 ]; thenshift $(($OPTIND - 1))printf "Remaining arguments are: %s\n" "$*"ARCHIVE_COMMENT="$*"elseprintf "Usage: %s: [-t target] [-e environment] comments \n" $0exit 2fiprintf 'Building %s - %s\n' "$CONFIG" "$ARCHIVE_COMMENT"# set Apple developer account as teamIDENTITY="" case "$CONFIG" inDEV) CONFIG="Debug"PROVISIONING_PROFILE="" PROVISIONING_PROFILE_NAME="" ;;QA) CONFIG="Ad-Hoc"PROVISIONING_PROFILE="" PROVISIONING_PROFILE_NAME="c" ;;UAT) CONFIG="UAT"PROVISIONING_PROFILE="" PROVISIONING_PROFILE_NAME="" ;;PROD)CONFIG="Production"PROVISIONING_PROFILE="" PROVISIONING_PROFILE_NAME="" # set Apple developer account as teamIDENTITY="" ;;*) echo "Configuration Option not valid, building for Production"CONFIG="Production"PROVISIONING_PROFILE="" PROVISIONING_PROFILE_NAME="" IDENTITY="" ;;esacPROJECT=`find . -name *.xcodeproj -print`if [ -z "$TARGET" ]thenTARGET=$PROJECTfixcodebuild -project ${PROJECT} -configuration ${CONFIG} -scheme ${TARGET} -destination generic/platform=iOS CODE_SIGN_IDENTITY="$IDENTITY" PROVISIONING_PROFILE="${PROVISIONING_PROFILE}" archive ARCHIVE_COMMENT="${ARCHIVE_COMMENT}" AUTOMATED_BUILD=YESARCHIVE_DIR=`ls -dt1 $HOME/Library/Developer/Xcode/Archives/*/*.xcarchive |head -n1`CURRENT_DIR=`pwd`IPA_DIR=`dirname $CURRENT_DIR`#create ipa signed binary packageif [ -f $IPA_DIR/$CONFIG.ipa ]thenrm $IPA_DIR/$CONFIG.ipafixcodebuild -exportArchive -exportFormat IPA -archivePath "${ARCHIVE_DIR}" -exportPath ${IPA_DIR}/${CONFIG} -exportProvisioningProfile "${PROVISIONING_PROFILE_NAME}"#uploadds response todeclare -i HTTPRESPONSEHTTPRESPONSE=`curl -X POST -sL -w "%{http_code}" http://testflightapp.com/api/builds.json -F file=@${IPA_DIR}/${CONFIG}.ipa -F api_token='' -F team_token=' ' -F notes="${ARCHIVE_COMMENT}" -F notify=True -F distribution_lists='DevOps' -o ${IPA_DIR}/${CONFIG}_curloutput.txt` echo "HTTP Response is" $HTTPRESPONSEcat ${IPA_DIR}/${CONFIG}_curloutput.txtrm ${IPA_DIR}/${CONFIG}_curloutput.txtif [ $HTTPRESPONSE -ne 200 ]; thenexit $HTTPRESPONSEfi
We do have the need to change identities and provisioning profiles because some of our users need to have the same app for the different environments loaded on the a single device at the same time, so they can make comparisons. And because the apps use push notification, each distribution requires its own unique app bundle id and provisioning profile.
We also need to change identities, because we use the enterprise program for development and in-house distribution and the Developer program for deployment to the app store.
The complete solution uses a TFS workflow that accesses the Mac Mini via ssh, copies the source code from TFS and runs the build script passing the environment, the target and the TFS PBI or Bug number and description as comments. The dev ops did their TFS magic there. I just helped on the Mac side to configure the proper ssh certificates, key pairs, proper keychain. On the TFS side, when the developer merges and commits the code to the QA branch, she has to associate it with a specific task or bug that this commit is supposed to fix. When queuing the build task, it is associated to the pbi or bug as well and additional information is captured from the user in a form. Between the form and the source code in the repository, the TFS build task finds out all the necessary information to copy the source code to the mac mini build server, run the build script and report the results back to the TFS. All the workflow is then controlled on the TFS/Windows side accessing the Mac Mini via ssh as needed.
4. Backend integration
Our apps access Salesforce on the backend most of the time. We have different end points on the Salesforce cloud depending on the environment we are targeting. Sometimes the end point is determined based on the credentials provided and other times the endpoint requires a change on certain URLs accessed by the app. The vendor that wrote the initial version of the app (Salesforce itself) opted for a conditional compilation of hard-coded constants in the pre-compiled header. They were first supporting only two sets of configurations (Debug and Ad-Hoc) and they would just use one #ifdef Debug to determine which set to use during build time. When I took over the coding, I had to support 4 sets. I added 2 more configurations to the project and each extra configuration sets its specific variable (DEV, QA, UAT, PROD).
I used all the conditional definitions that I could inside the build settings. I have specific bundle IDs based on conditional settings provided by build settings in Xcode. I have specific app name and display name based on Executable_Prefix that is conditionally defined in build settings as well.
My next step will be to remove the hard-coded constants from the pre-compiled header and put them into a plist file like the UA configuration for push notification. In the future, we can have in the future an API that allows us to change these settings at runtime, without the need to submit upgrades to the App Store for approval, when we change our endpoints on the backend.
5. Corporate network
This was one of the hardest things to work around, because the security restrictions and the lack of privileges developers have in our network. For the TFS build workflow to work, we had to create a local user on the Mac Mini that was used to host the developer's certificate and signing authority certificates in its keychain that we could then unlock for the xcodebuild to access during building and archiving.
The service domain user prepares all the files for the build and switches to the local user to execute the build script. The local user only exists on the build server and it has minimal permissions on the box.
We had to do that in order to avoid sending sensitive passwords over the ssh session or inside the scripts. Even though we were authenticated on the ssh session, OS/X was still asking for a password to allow xcodebuild access to the keychain. We tried setting this access for xcodebuild in the keychain, but it didn't work.
The upload to TestFlight part of the script would fail every other try due to proxy server authentication problems. The dev op working with me found out that the cause was a bug on Mavericks when the Mac is connected to a gateway with redundancy (see this discussion here). We added a USB to RJ45 adapter and switched to using this NIC to solve the problem. Now, even the VNC sessions with the Mac Mini work much better.
6. Maintaining the version string consistent across deployments
More than once, when I was hasting the manual deployments, I forgot to update the version short and long strings accordingly.
First, I think Apple calling the short version string "version" and the long version string "build" was a mistake that causes a lot of confusion. TestFlight shows the releases of your builds as () but based on Apple's documentation, what they call build is in fact . It doesn't have to be that way, you can treat both of them independently, but that's what is expected. And when you submit an update for your app to the apple store, they always expect both of them to be higher than the previously submitted binaries.
What the vendor did on their code was to keep both identical and manually increment both together when making a new deployment.
After a couple of mistakes in the manual process, I decided to incorporate the increment of these numbers to the archive script.
We settled for a short version string in the format of M.m.r (major. minor.review) and the build or long version string as M.m.r.build.
Now, every time we run a QA archive the script increments the build number, and every time we deploy a production build, the script increments the review number.
The major and minor numbers don't change that often, so they are still maintained manually.
Conclusion
I'm also starting to use configuration files that I can load in Xcode across different projects. I'm compiling all the different build settings from all our projects, determining the common settings, the unique settings per environment and per project that I can segregate in separate configuration file and then refer to them in Xcode.
The build script right now is project dependent. I'm also working on defining project templates or wizards that can be used by other developers when starting a project from scratch. I'm still considering if I should do it like wizards that capture the project specific definitions from a user dialog and then create the project tree for you, a template that the user clones to build his own project or a tool that traverses the project and builds the necessary scripts based on the projects definitions.
The solution is far from complete and I'm still learning the best way to use each tool. I'm also hopeful that in the near future my company will upgrade TFS to fully support git. Until then, this combination of scripts, programs and tools will help us deploy faster and better than doing everything manually.
If you have suggestions to help us, they are all welcome. Please, leave your suggestions in the comments area, or share with us your solutions to similar problems.
