Friday, March 6, 2020

Book review - A Elite do Atraso, by Jessé Souza

I really enjoyed  the book "A Elite do Atraso", by Jessé Souza. 
For the ones that don't know him, here is a wkipedia stub (https://en.wikipedia.org/wiki/Jess%C3%A9_Souza). He is a Brazilian thinker who has done extensive research in the field of Social Theory. He accurately captures our current state of affairs. Although his focus is on Brazilian society, most of his research is applicable to the United States and to other countries around the world. He clearly shows some bias at times, but he acknowledges that and it is not hard to see beyond it.  

His book "A elite do atraso" is fascinating reading. According to him, at some point in the mid 1970s, we started the shift from industrial capitalism to finance capitalism. At this point, the fiscal state, which had effective mechanisms to tax the industrial transactions, became a deficit state. While it's very costly to shift industrial operations across national borders, moving financial assets is very easy and very low cost. Miraculously, the tax income disappeared and states had to borrow money to pay for the needs of society. By converting their industrial operations into financial assets, the elite not only avoided taxation, but became the lenders of the states. They stopped paying taxes via financial manipulations. This reduce the state's revenue, which needed then to borrow money to pay for its expenses. The lenders were the ones that retained the money. By charging interest on the money they previously owed to the state, they were also able to accumulate more economic capital at even faster rate. The state lost revenue and increased the expenditure in the form of interests paid on the debt accrued.

Souza  points out the mistake of using income or amount of economic capital accumulated to classify society in upper, middle and lower classes. There is a psychological and emotional aspect grouping individuals in a society that transcend their pure economic capital.

Instead, he prefers to analyze society based not only on the economic capital, but also on their abilities and sentiments and their access to other types of capital, like cultural capital. It seems that he got some inspiration on the work of Pierre Bourdieu, but he expanded way beyond Bourdieu's views. I'm no expert on Souza or Bourdieu, but as far as I could read from both, Bourdieu focus seems to be on education and its importance in changing the predicted outcome of an individual. Souza expands on Bourdieu work, but explaining better why the cultural capital has this effect on the outcome of individuals and which individuals in our society have access to this type of cultural capital.

For Souza, the middle class is the extract of society that is is responsible for the perpetuation and propagation of our societal model, assuming the tasks of both controlling and supervising material goods and justifying and legitimating the social order, both at market and state levels. The middle class has limited access to economic capital, but they have access to cultural capital and their ability to accumulate cultural capital is what gives them a slim chance of ascending to the higher echelons of society. This possibility, albeit slim, is what keeps them faithful to their role in society. Also, the impression that the accumulation of cultural capital is result of their merit in pursuing it, gives the middle class the perception that their cultural capital accumulation is derived from their merit only and not privilege. They see themselves as the moral compass of society.

He divides the middle class into 4 distinct groups: Protofascists (30%), liberals (35%), expressives (20%) and critics (15%).

The first two groups are the ones more traditionally recognized as middle-class, the class of the technical knowledge the directly serves the capital needs and its reproduction and less capable of prompting social transformation.

Protofascists are the ones that embrace hate and spend very little effort in reflecting upon facts and opinions. They are usually very Manichaeans, with a simplistic and clear division between good and evil. 

Liberals share the same simplistic, moralistic view of the world, but for them the democratic rituals establish an organized existence for all, and they need some convincing arguments to accept exceptions to the democratic rule. We can say that for this group, the ends might justify the means, even if these means are questionable but the ends are of high moral ground. At this point I disagree with the author's view that this is a singularity of the Brazilian society. The election of Donald Trump in United States, strongly supported by the conservative, religious middle class, despite his questionable morals, proves that both countries share the problems. There are points along the book in which the author lists certain feature unique to Brazilian society that are easily applicable to other countries. 

Moral, for Souza has two facets: one is of productivity and another one is creativity. While the first two groups positioned themselves on the productivity side of morality, the expressives embrace the creativity, which he equates to the ability to be faithful to one's deepest emotions and feelings. He calls them expressives, because they are the ones that harness the ability to express these feelings and emotions that are usually repressed in favor of productivity.

One can be "expressive" and unable or unwilling to make any social critique that can promote effective social change. We can say that this group has the heart in the right place, but has a problem of prioritizing the solutions. In face of saving the whales and lifting millions out of poverty, they have a hard time deciding which should come first for society. They would probably go for saving the whales in hopes that this will cause enough social change to allow the poverty problem to solve itself. For me this would be the ostrich group, that buries its energy in less socially controversial issues in hopes that they can pull the solutions for the more controversial issues in their wake. 

On the other end of this spectrum of the groups with higher cultural capital inside the middle class are the critics. They are the ones that perceive the social world as a construct and hence can be subjected to change. This is a counterpoint of the social world being a given reality that one has to adapt to. This group bears the burden of being conscious of the contradictions demanded of the middle class and realizing how hard is to achieve personal and social freedom in the context of a wicked and oppressive society. It sounds like a romantic view to me.

I'm wondering if all the empirical data the author collected for his research captured the incidence and type of mental illnesses on each group. They all seem to present a pathological behavior to me and I bet the Critics have a high incidence of depression.

I'm almost done with the book. It's not an easy reading. The author definitely didn't target engineers when writing it. 

Also, I believe he is very passionate about the recent events in Brazilian politics, like the impeachment of the first female president of Brazil and the dismantling of successful government programs to lifted millions out of poverty. I don't know if he didn't have data, resources or both to compare his empirical findings in Brazil with that of other countries, which made him conclude that his models reflect some unique characteristics of the Brazilian society. References he makes of researchers from other countries lead me to believe that maybe he didn't want to make assertions about other countries without analyzing a larger dataset. I hope he does that and I also hope to find his publications in English, because I think his work is deserving a broader audience and ample discussion.

While reading his book I tried to check his conclusions against other sources of information and found some other references about ways to stratify the middle class that have hints of his take: https://www.pewresearch.org/wp-content/uploads/sites/3/2010/10/Four-middle-classes.pdf In this paper  by Richard Morin, Senior Editor, Pew Research Center they tried to expand the concept of middle class beyond the income per se and some of their conclusions are similar to the ones by Souza. They also divided the middle class into 4 groups and the percentages are very similar to the ones found by nSouza. It would be interesting to apply the same clustered analysis they employed to the data collected by Souza to see how similar or dissimilar the middle class is in both countries. It would also be interesting to read the scientific paper produced by Souza that originated his book for the broader public.  




Wednesday, January 1, 2020

How much is my behavioral surplus worth?

One thing that concerns me is how much money big players in the surveillance capitalism are making on the behavioral data they mine from all of us and the majority of the people are not benefiting from it.

Sure there is value on all the research they've done. Today we have better weather prediction, disease control, emergency response, and on and on, but they are also feeding our information to rogue players that are plainly exploiting the population at large with the prediction models they created.

For example, take the insurance industry. They make huge profit on reducing their risk using the prediction models. Today, any kind of insurance we buy, be it car, home warranty, appliances, life, health insurance is crafted in a way that everyone insured will lose money and the insurance carriers will always profit big time.

The big behavioral surplus miners, like the Googles, FB, big retailers, big media corporations have an impressive power and share a slice of it with the willing customers like big corporations in several industries: pharmaceutical, finance, insurance, retail, so they can prey on the consumers and individuals.

Consumers and individuals are constantly fragmented, segmented, so they never can amass enough cohesion to respond to this assault. 

In my point of view, it's fruitless to prevent, reduce or contain the behavioral surplus mining. The genie is already out of the bottle. What we can do is to reduce the disparity between the players, by providing the consumers and individuals with a transparency about these predictions models. 

For example, a senior citizen is coerced into buying warranty for the tablet she is acquiring at the retail store to have video calls with her grandchildren. In absolute numbers the values seem to be minimal, but for a senior citizen on a fixed income these small assaults add up. She might be lucky and one of her children or one of her grandchildren will be around and will do the math and show her that it is not worth it, that the retailer is only pushing the warranty product on her because of the high profit margin of this product. For the retailer the tablet is just a bait to sell the warranty. Phone carriers do a similar thing when they "offer" free tablet with your phone plan and then shove the monthly data usage fee and the warranty for it, because "you don't want to lose or get the tablet damaged and not being able to replace it for free".

The point is, not everyone has a smart daughter or granddaughter handy when we make our buying decisions. And for society as a whole, wasting our buying power into these fictitious products is preventing us from putting money in other actual products that are more beneficial for society. They work as money concentration tools.

We need an impartial agent that researches and provides AI agents that can be accessed by everyone when making spending decisions, with the intent for everyone to be better off in the long run. This will level the plain field and will make both sides of the transactions better in the long run.

In my opinion, the most efficient way to create such impartial agent is to use the government for that, but when the government is held hostage of mobsters, we need to find alternative agents that are willing to work for the common good. Maybe a consortium of marketing research companies can step up to to plate and create the proper tools for the common people to police the behavioral surplus mining or to benefit from big data in the same way big corporations do. Why marketing research companies? Because they are also losing in the big data mining industry. The big players control the sensors that are the fringe of the network, in the hands of the consumers, the smart phones and the encrypted popular social network sites that aspirate all the behavioral surplus from everyone. Its becoming harder and costlier for marketing research companies to acquire the important data that allow them to do their job. Google, Facebook, Amazon, Microsoft, they are all rendering marketing research companies obsolete. It is hard to convince a consumer to wear an additional meter besides just my smart watch or my smart phone, just so the marketing research firm can collect some approximation of the data that is actually being collected by the manufacturer of the smart device or its operating system. And in the process of developing such autonomous data collecting tools, by using public diagnostic APIs and cloud based tools, the marketing research companies provide even more information for the big players, that also control the software development tools market.

Here is an opportunity for such companies to re-invent themselves as stewards of the surveillance capitalism, bringing order and information justice to society and keeping themselves viable in the future.


Thursday, December 12, 2019

Time-tracking tyranny

Why do corporations bestow so much power into middle management?

In the age of agility and increased productivity, it makes no sense to have so much leverage in the hands of non-productive people.

In my point of view, if you have a team full of productive folks, for example, in a software company, a bunch of software engineers and programmers, you will eventually produce something usable. But if you assemble a team only with managers, that have no idea on how to write code, you will never be able to produce anything.

Of course I accept that with good management we can produce something maybe faster, maybe better, but still, who writes the code is who is producing the solution, not the manager enabling it. So it makes no sense to me that so much power resides in middle management.

Middle management sits a filter layer between the executives and sponsors. When AI, machine learning, pervasive data analysis tools were not available, it's understandable that we need this filter, so the executives can get a grasp of is going on at the ground level.

Today this middle management filter is just distorting the view of what happens on the ground with their personal biases and lack of technical knowledge.

We lose agility, stifle innovation and inhibit the flow of information. Middle management act as bottleneck of collaboration, with personal insecurities, their anxiety into the power play of corporate politics and their need to be "in the know", to be copied in all communications, to be invited to all meetings....

Here is where timesheets come in as a torture mechanism impinged on the actual workers. Every single corporation I worked for puts so much importance on employees filling timesheets and it looks like the world is going to crumble if we don't do so ahead of time all the time.

Funny thing is that the time workers put into their jobs can be extracted from their actions publicized in other tools of the company: emails, calendar entries, meeting participation, source code commits, product deployments, browsing activity, network activity... Corporations have plenty of surveillance tools in place that can precisely identify when and where an employee have been all the time. If it is a legal requirement that the employee acknowledges the time worked, why not produce the time sheet with the surveillance information, using certain guidelines, like certain workers are not entitled to overtime, etc, and present this pre-filled timesheets to the employees for their approval. Machine learning algorithms can be used to make predictions for future entries if there is a need to estimate future time that will be spent on a product for planning purposes.

The same way corporations got rid of a whole slew of secretaries, phone operators, office clerks, by automating tasks, empowering other workers with smarter tools, I think it's now the time to re-think the need for middle managers.

Sunday, July 23, 2017

The past is an outfit that doesn't fit us anymore

I know a lot of Bernie Sanders followers are going to execrate me for this post and and I might be on the wrong side of a lot of right people the I admire, but Bernie and to some extent Hillary, were like a Jodorowski movie or supporting Lula and the Brazilian Labor party in 2017.  They are our grandparents or our aging parents that tell us their stories of their youth, their recounted past that time had smoothed the rough edges and it is now exciting the kids to give it a try.

In a sense, they highjacked the thirsty for change of the younger generation, piggybacked on the new crowd technology, reviving their bruised ego to give it another chance in history.

I decided to support Hillary, because at least having a woman in the white house would be a big change. Instead, the electoral college decided to go for another big change and elect a shrewd crook with dishonesty and authoritarian rule unheard of in our history.

It's a mistake to think that the revolutionary work done by Hillary, Bernie and their generation has a lot in common with Occupy Wall street. Failing to recognize the differences is what defeated the progressive view they wanted to win.

The same way in Brazil, the country definitely owns a lot to the relentless work done by Lula and the Labor party, but their past shouldn't forgive the criminal acts that were performed with their knowledge, or participation, or connivence. Because if this is fair, then it is also fair to forgive the trespasses of the Cardosos, the Neves, the Temers, etc. They were all affected and fought one way or another during the military dictatorship. They all did good things for the country and later on, when drowning on the inebriation of the power, got corrupted. Forgive them and give them another chance is not the solution.

I know it seems that now the world is beyond a rational solution, but guess what, the same organ the we use for reasoning is the one responsible for our feelings and until we not only recognize feelings as part of our reasoning but we learn how to acknowledge and use them as such, we will be stuck in this collective cognitive dissonance.

To the Jodorowski fans out there, I'm among them. I supported his work in crowdfunding platforms, I treasure my Jodorowski dollars and my movie posters. I smile with warm feelings watching his movies, but it is not for their innovative aesthetics, but for the nostalgia of the time in my teenager years when this type of aesthetics was cutting edge, never seen before. I enjoy seeing the hidden unintended meanings I can get by admiring how his son engages with him, participating on his work. How sweet and ludic their relationship seems to be on the screen. He aged much better than Woody Allen, for sure. But today's indie movie scene has a lot better and more enticing story telling that appeal to the younger generations. His movies look like the old pair of jeans that I found in the attic that still fit, but look undeniably old. They might fit on me, but I am the one that don't fit in them anymore.










Thursday, April 21, 2016

Extreme decluttering in baby steps

People that know me can testify about how attached I am to my things, specially books. The problem is that, as of late, the entropy level at home is just so high that things are getting unmanageable. I'm on the verge of becoming what most people qualify  as a hoarder.

So I had to take drastic measures. I'm giving away several books:
  • Old computer books about older versions of Java
  • Computer books about waterfall methodology
  • Feng-Shui books
 That's it. A big milestone for me. Finally I'm able to admit to myself that my house is beyond any possibility of Feng-Shui transformation and no benefit can be achieved by my family or myself from me struggling with guilt every time I have to move these Feng-Shui books out of the way when I'm looking for something that has been lost in the huge pile of mess that is any horizontal surface in my house.

Not that eliminating ALL Feng-Shui books will make any significant dent in the havoc that surrounds my life right now. But it is a step towards reconciliation with myself. A baby step, I know, but still a step forward.

Thursday, April 9, 2015

Hiring talents, the Walmart conundrum

Monday my boss called a meeting between UX guys and the mobile developers and him to decide on the new feature of capturing multipage document.I was the only one that got to the meeting with one suggestion of UI flow and coding solution that would require no webservice change at all.I made a drawing on the whiteboard and I said I could make a mockup.he told us to think about it and asked the UI guys to come up with ideas too.2 h after the meeting I had a small prototype, already integrated with the app, that they could test out the UI of my idea.The next day, I had a fully functional prototype of the solution, including the conversion of all the pages in a single pdf file.I sent the workable prototype for them to install on their devices and my boss replied to my message asking the other two mobile developers to send their mockups as well.He called another meeting for today and he started the meeting by showing my prototype on his phone and he started asking for small changes here and there and assigning tasks here and there when I told him I had two other mockups to show.Again, I showed two working apps with a completely different flow, that were even better than the first one and that blew his mind...Considering that I got two production bugs to fix and I also fixed 3 other bugs that QA didn't get but I found in production and I fixed all of them as well, not bad for a week.

When a company wants to get into a new technology and they don't have the expertise in that area, they think that the best way is to hire the best talent possible.but they don't have enough knowledge to evaluate if the one showing oneself as the best is indeed the best at the same time they feel insecure for not being able to grasp everything the expert is providing.then comes the bitter feeling in the company that the expert is too smart for their liking.

If they hire the expertise from 3rd party vendors, the integration with the current culture fails. It's almost impossible to satisfy them, because they don't know what questions to ask and are giving several answers to grade when the product is delivered. Or if they hire someone like me, the expert ends up doing guerrilla development, because I have to constantly sneak new ideas and push them forward from behind the lines, making obscure alliances, or being ostracized, unmotivated up to the point that both parts decide to split.

Thursday, July 17, 2014

My quest to automate Mobile deployments - How to prompt for user input during Xcode builds

At work, as the sole mobile developer in the office, I have to juggle between writing code, understanding 3rd party code that was acquired by the company and also deploying the code (mine and 3rd party). I had to upload binaries to TestFlight for QA testers, for show and tell to executives, manage iTunesConnect deployments, at the same time I have to keep the local git repository in sync with the corporate TFS repository.

As it happens, my company uses TFS that is not git friendly yet. I quickly found git-tf to bridge the two repositories, but the job of manually syncing the local and remote repositories all the time and all the manual deployments were taking up too much of my development time.

Another thing that was making the deployments complex was the amount of changes I had to make in the configuration and the code when migrating from one environment to another.

So I quickly started working on a solution to automate my tasks as much as possible. Not because I'm lazy, but mainly because if we write the proper automation tasks, we avoid mistakes.

Here are my challenges:

1. Push notifications

Our apps use push notification and we use Urban Airship as our push notification server. As is well known, UA and APNS (Apple Push Notification Server) have two environments: Debug and Production. UA provides a configuration file where you can put the keys for both of these environments and at run time, it configures itself accordingly. The problem is that at my company, we have to handle 4 environments: Development, QA, User Acceptance Test and Production. All the pre-production environments have to use APNS debug, since they are not production, so I had to handle 3 different debug configurations for UA config file.

2. TFS integration

All the "other" code in the company is developed in .NET world. They get along quite well with TFS, SQL server, Visual Studio, Windows, but when it comes to git and Macs, the support is questionable. The company didn't like the idea of having a separate git server for mobile developments. They prefer the code to be imported into TFS and from there they can merge/branch according to the deployment environment.

As they have a lot invested in and knowledge of the TFS infrastructure, they want to benefit from it as much as possible.

3. Lack of Macs

Due to the cost of Macs, the company prefers to have mostly Windows workstations and laptops handed out to developers. We have very few Macs and usually they are handed out to executives that run a Windows VM on it all the time. They conceded that developers of iOS code need a Mac, but it was hard for them to accept that all dev ops would need a Mac too in order to build and deploy code to TestFlight and App Store. Having a Mac there is a sign of status that they were just not ready to spread to all levels of development and IT tasks.

4. Back-end integration

Our apps talk to Salesforce APIs on the back-end and again, each environment requires different endpoints and sometimes slightly different state machine to entertain the SOAuth dialog and certain customizations in the login process, like allowing the user to reset password from the login dialog ("forgot password link") and the always connected app.

5. Corporate network

My company is in the financial sector, so security is very strict. The network has all types of firewalls, traffic filters and restrictions they put in place for all users. The users are defined in Active Domain and the network is very Mac hostile. We have to change our passwords very often and every time it takes place, it is a nightmare on the Mac side. I have to either clear all sorts of cookies, certificates and passwords stored on the keychain and start over, entering the new password again a thousand times, or I have to painstakingly go through my key chain and update all the stale passwords.  If you add to that the bug on Mavericks when your Mac is connected to redundant gateways, which disconnects every minute or so, remote tasks are very hard to accomplish on Macs here.

6. Maintaining the version string consistent across deployments

Uploads to the App Store require that the version strings (long and short) increase from one release to another. We also need to be able to identify uniquely each bundle uploaded to TestFlight or installed on a device. We need to look at the app and quickly identify which distribution is installed and which build generated that distribution. 

Solutions:

Just trying to make my life easier, I started looking into ways to simplify my tasks with automation.

1. Push Notification

For the push notification, I wrote script steps into the build action on the scheme that updates the UA config file before the archive.
Here is an example:

# Archive pre-actions  
# set UA config to development
/usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"
if [ "$CONFIGURATION" == "Ad-Hoc" ]; then
    /usr/libexec/PlistBuddy -c "Set :CFBundleVersion $buildNumber" "${PROJECT_DIR}/${INFOPLIST_FILE}"
    /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_KEY " "${PROJECT_DIR}/AirshipConfig.plist"
    /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_SECRET " "${PROJECT_DIR}/AirshipConfig.plist"
    /usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"
fi
if [ "$CONFIGURATION" == "Production" ]; then
echo "actions before archiving for Production upload" >> ~/Public/${CONFIGURATION}MobilePartnerarchive.txt
# change UA config
    /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_KEY " "${PROJECT_DIR}/AirshipConfig.plist"
    /usr/libexec/PlistBuddy -c "Set PRODUCTION_APP_SECRET " "${PROJECT_DIR}/AirshipConfig.plist"
    /usr/libexec/PlistBuddy -c "Set APP_STORE_OR_AD_HOC_BUILD YES" "${PROJECT_DIR}/AirshipConfig.plist"
fi

In this example, both ad-hoc and production of an enterprise program app use UA/APNS production, but they have different app bundle ids, which means they need to point to different topics in UA/APNS server,. For that reason, I need different keys/secrets for each distribution. In this example, we can populate the keys and secrets right before the archive, maybe using a script that captures them from the keychain on the build machine using the dev ops build user. The developer will never have access to these keys/secrets on her keychain.

2. TFS integration

For TFS integration I installed git-tf on the Mac. It is tricky to configure all the required java stuff and the symlinks to make it easy to update the versions in the future. You have to clone the TFS repository on the command prompt and if you don't configure the clone afterwards to store your credentials on the git configuration file, you will have to provide them every time you execute any git-tf command. You also have to create your .gitignore, to avoid grabbing all the Xcode temporary files into your TFS repository. 

I couldn't find a way to call these commands from within Xcode when I'm in the source control menu. I think Xcode should provide a way to customize the actions there, similar to the schemes, providing hooks for pre- and post- action. This would have made my life a lot easier. You won't believe how many iOS and Windows developers are not familiar with command prompt in the terminal window.

The other bothersome feature was the fact that configuring git-tf to store the credentials would store the password unencrypted, and this configuration had to be changed manually every time we had to change our passwords in AD.

I started by writing scripts that would store the password on the keychain when cloning the repository and then would read the key chain to provide the password when checking in code to TFS or pulling in updates. They were still scripts that would run on the Terminal window and the developer had to remember to checkin code to TFS when she was ready to merge to other branches.

This also didn't solve the problem of automatically changing the password when the user had to change the password in Active directory. I'm still thinking about how to solve this problem of the password update triggered by AD.

The next step was to make the checkin to happen from within Xcode, without the need to run any scripts in the terminal window.

I usually checkin the code when I'm ready to deploy a binary to TestFlight, which means that I had generated an archive. As it happens, sometimes I would upload a binary to TestFlight and I would forget to checkin code to TFS, which would make the TestFlight code out of sync with TFS. I then decided to create a post archive script to checkin to TFS every time I generate an archive. 

Unfortunately, git-tf does not return an error if the connection with TFS fails due to invalid credentials. In fact, if we keep providing an invalid password, it keeps asking for the password again 4 times before it ends without returning any error to the calling script. Which means that it can even lock the account depending on the policies of your company.

Since I didn't resolve the changing password problem, I decided to add a script that would always prompt the user for a password. At least the developer doesn't need to go to the terminal to run a script to checkin.  Because I don't know how to show a prompt in Xcode without using AppleScript, that's the language I had to use to accomplish that.

Here is an example:

set automatedBuild to do shell script "echo ${AUTOMATED_BUILD:='NO'}"

if (automatedBuild ≠ "YES") then
    set archiveComment to "Xcode generated"
    set dialogResult to display dialog "Enter TFS password:" default answer "" hidden answer yes buttons {"OK"}
    set userName to text returned of dialogResult
    do shell script "/bin/bash --login -c \"cd $SRCROOT/.. ; echo "& userName & " | git-tf checkin \" 2>&1  "
    display dialog "Code saved to TFS..." buttons {"OK"}
end if

set archiveDir to do shell script "ls -dt1 $HOME/Library/Developer/Xcode/Archives/*/*.xcarchive |head -n1"
try
do shell script "/usr/libexec/PlistBuddy -c \"Add :Comment string " & quoted form of archiveComment & "\" " & quoted form of archiveDir & "/Info.plist"
on error

end try
do shell script "/usr/libexec/PlistBuddy -c \"Set :Comment " & quoted form of archiveComment & "\" " & quoted form of archiveDir & "/Info.plist"
do shell script "/usr/libexec/PlistBuddy -c 'Save'  " & quoted form of archiveDir & "/Info.plist"
do shell script "/usr/libexec/PlistBuddy -c \"Set :Name 'QAbuild'\" " & quoted form of archiveDir & "/Info.plist"
do shell script "/usr/libexec/PlistBuddy -c 'Save' " & quoted form of archiveDir & "/Info.plist"

This script assumes that the user has configured git-tf with her username, but it is not hard to change it to get the user name from the user running the archive in Xcode.

This code also changes the comment and the name of the archive file, so we can see on Xcode Organizer that this build was generated by Xcode and not by our automated build script.

I have the test of the Automated Build flag because the same scheme is used in our automated build solution that I'll explain later.

3. Lack of Macs

As I explained, Macs are hard to get at my company. But the fact that all the code I was working with had to be built on a Mac and that I was the only person with access to it, convinced my boss to invest in a Mac Mini server to be our build server. I had now to make it possible to remotely trigger the build and deployment on that machine.

My first try was using Bots. It took me awhile to figure out how to use them. I demo'ed to my boss and he was very excited, because the dev ops could access the bots via web from their Windows boxes.

The major road block with Bots is that it requires git in order to work. I also didn't find the Xcode documentation very enticing and it lacks more meaningful examples. I did entertain the idea of scripts that would pull the code from TFS on the a git server on the build machine, but I haven't convinced dev ops of the feasibility of this solution yet. They were already developing workflows in TFS that queue the builds of all the .NET stuff and they opted for going that way.

I wrote a script that receives as arguments the environment of the build (DEV, QA, UAT, PROD), the scheme (I have some projects with more than one target) and it sets the identities, provisioning profiles, etc for xcodebuild to generate the archive directly from the command line. It also generates the .ipa file and uploads it to TestFlight if the build is for a non-production environment. The idea is to extend this build script to upload the binary to the Appstore for production environments of our public apps, but we are not quite there yet. This will remove the need for the dev ops to access a Mac except running remote tasks from Windows or accessing web servers.

Here is the example of the build script:

while getopts t:e: option
do
case "${option}" in

e) CONFIG=${OPTARG};;
t) TARGET=${OPTARG};;
?|*) printf "Usage: %s: [-t value] comments \n"  $0
exit 2;;
esac
done

if [ $OPTIND -gt 1 ]; then
shift $(($OPTIND - 1))
printf "Remaining arguments are: %s\n" "$*"
ARCHIVE_COMMENT="$*"
else
printf "Usage: %s: [-t target] [-e environment] comments \n"  $0
exit 2
fi

printf 'Building %s - %s\n' "$CONFIG" "$ARCHIVE_COMMENT"

# set Apple developer account as team
IDENTITY=""

case "$CONFIG" in

DEV) CONFIG="Debug"
PROVISIONING_PROFILE=""
PROVISIONING_PROFILE_NAME=""
;;
QA) CONFIG="Ad-Hoc"
PROVISIONING_PROFILE=""
PROVISIONING_PROFILE_NAME="c"
;;
UAT) CONFIG="UAT"
PROVISIONING_PROFILE=""
PROVISIONING_PROFILE_NAME=""
;;
PROD)CONFIG="Production"
PROVISIONING_PROFILE=""
PROVISIONING_PROFILE_NAME=""
# set Apple developer account as team
IDENTITY=""
;;
*) echo "Configuration Option not valid, building for Production"
CONFIG="Production"
PROVISIONING_PROFILE=""
PROVISIONING_PROFILE_NAME=""
IDENTITY=""
;;
esac

PROJECT=`find . -name *.xcodeproj -print`

if [ -z "$TARGET" ]
then
TARGET=$PROJECT
fi

xcodebuild -project ${PROJECT} -configuration ${CONFIG} -scheme  ${TARGET} -destination generic/platform=iOS CODE_SIGN_IDENTITY="$IDENTITY" PROVISIONING_PROFILE="${PROVISIONING_PROFILE}" archive ARCHIVE_COMMENT="${ARCHIVE_COMMENT}" AUTOMATED_BUILD=YES 

ARCHIVE_DIR=`ls -dt1 $HOME/Library/Developer/Xcode/Archives/*/*.xcarchive |head -n1`
CURRENT_DIR=`pwd`
IPA_DIR=`dirname $CURRENT_DIR`

#create ipa signed binary package

if [ -f $IPA_DIR/$CONFIG.ipa ]
then
 rm $IPA_DIR/$CONFIG.ipa
fi

xcodebuild -exportArchive -exportFormat IPA -archivePath "${ARCHIVE_DIR}" -exportPath ${IPA_DIR}/${CONFIG} -exportProvisioningProfile "${PROVISIONING_PROFILE_NAME}" 

#uploadds response to
declare -i HTTPRESPONSE

HTTPRESPONSE=`curl -X POST -sL -w "%{http_code}" http://testflightapp.com/api/builds.json -F file=@${IPA_DIR}/${CONFIG}.ipa -F api_token='' -F team_token='' -F notes="${ARCHIVE_COMMENT}" -F notify=True -F distribution_lists='DevOps' -o ${IPA_DIR}/${CONFIG}_curloutput.txt`

echo "HTTP Response is" $HTTPRESPONSE
cat ${IPA_DIR}/${CONFIG}_curloutput.txt
rm ${IPA_DIR}/${CONFIG}_curloutput.txt

if [ $HTTPRESPONSE -ne 200 ]; then
exit $HTTPRESPONSE
fi

We do have the need to change identities and provisioning profiles because some of our users need to have the same app for the different environments loaded on the a single device at the same time, so they can make comparisons. And because the apps use push notification, each distribution requires its own unique app bundle id and provisioning profile.

We also need to change identities, because we use the enterprise program for development and in-house distribution and the Developer program for deployment to the app store.

The complete solution uses a TFS workflow that accesses the Mac Mini via ssh, copies the source code from TFS and runs the build script passing the environment, the target and the TFS PBI or Bug number and description as comments. The dev ops did their TFS magic there. I just helped on the Mac side to configure the proper ssh certificates, key pairs, proper keychain. On the TFS side, when the developer merges and commits the code to the QA branch, she has to associate it with a specific task or bug that this commit is supposed to fix. When queuing the build task, it is associated to the pbi or bug as well and additional information is captured from the user in a form. Between the form and the source code in the repository, the TFS build task finds out all the necessary information to copy the source code  to the mac mini build server, run the build script and report the results back to the TFS.  All the workflow is then controlled on the TFS/Windows side accessing the Mac Mini via ssh as needed.

4. Backend integration

Our apps access Salesforce on the backend most of the time. We have different end points on the Salesforce cloud depending on the environment we are targeting. Sometimes the end point is determined based on the credentials provided and other times the endpoint requires a change on certain URLs accessed by the app. The vendor that wrote the initial version of the app (Salesforce itself) opted for a conditional compilation of hard-coded constants in the pre-compiled header. They were first supporting only two sets of configurations (Debug and Ad-Hoc) and they would just use one #ifdef Debug to determine which set to use during build time. When I took over the coding, I  had to support 4 sets. I added 2 more configurations to the project and each extra configuration sets its specific variable (DEV, QA, UAT, PROD).

I used all the conditional definitions that I could inside the build settings. I have specific bundle IDs based on conditional settings provided by build settings in Xcode. I have specific app name and display name based on Executable_Prefix that is conditionally defined in build settings as well.

My next step will be to remove the hard-coded constants from the pre-compiled header and put them into a plist file like the UA configuration for push notification. In the future, we can have in the future an API that allows us to change these settings at runtime, without the need to submit upgrades to the App Store for approval, when we change our endpoints on the backend.

5. Corporate network

This was one of the hardest things to work around, because the security restrictions and the lack of privileges developers have in our network.  For the TFS build workflow to work, we had to create a local user on the Mac Mini that was used to host the developer's certificate and signing authority certificates in its keychain that we could then unlock for the xcodebuild to access during building and archiving.

The service domain user prepares all the files for the build and switches to the local user to execute the build script. The local user only exists on the build server and it has minimal permissions on the box.

We had to do that in order to avoid sending sensitive passwords over the ssh session or inside the scripts. Even though we were authenticated on the ssh session, OS/X was still asking for a password to allow xcodebuild access to the keychain. We tried setting this access for xcodebuild in the keychain, but it didn't work. 

The upload to TestFlight part of the script would fail every other try due to proxy server authentication problems. The dev op working with me found out that the cause was a bug on Mavericks when the Mac is connected to a gateway with redundancy (see this discussion here). We added a USB to RJ45 adapter and switched to using this NIC to solve the problem. Now, even the VNC sessions with the Mac Mini work much better.

6. Maintaining the version string consistent across deployments

More than once, when I was hasting the manual deployments, I forgot to update the version short and long strings accordingly. 

First, I think Apple calling the short version string "version" and the long version string "build" was a mistake that causes a lot of confusion. TestFlight shows the releases of your builds as () but based on Apple's documentation, what they call build is in fact .  It doesn't have to be that way, you can treat both of them independently, but that's what is expected. And when you submit an update for your app to the apple store, they always expect both of them to be higher than the previously submitted binaries.

What the vendor did on their code was to keep both identical and manually increment both together when making a new deployment.

After a couple of mistakes in the manual process, I decided to incorporate the increment of these numbers to the archive script. 

We settled for a short version string in the format of M.m.r (major. minor.review) and the build or long version string as M.m.r.build.

Now, every time we run a QA archive the script increments the build number, and every time we deploy a production build, the script increments the review number. 

The major and minor numbers don't change that often, so they are still maintained manually.

Conclusion

I'm also starting to use configuration files that I can load in Xcode across different projects. I'm compiling all the different build settings from all our projects, determining the common settings, the unique settings per environment and per project that I can segregate in separate configuration file and then refer to them in Xcode.

The build script right now is project dependent. I'm also working on defining project templates or wizards that can be used by other developers when starting a project from scratch. I'm still considering if I should do it like wizards that capture the project specific definitions from a user dialog and then create the project tree for you, a template that the user clones to build his own project or a tool that traverses the project and builds the necessary scripts based on the projects definitions.

The solution is far from complete and I'm still learning the best way to use each tool. I'm also hopeful that in the near future my company will upgrade TFS to fully support git. Until then, this combination of scripts, programs and tools will help us deploy faster and better than doing everything manually. 

If you have suggestions to help us, they are all welcome. Please, leave your suggestions in the comments area, or share with us your solutions to similar problems.