Vijay Talks Tech

macOs grep -f Is a Nested Loop Join; Use ripgrep or mawk Instead

2026-08-06T09:00:00-07:00

4887x: the same job through the grep macOS ships and through ripgrep, a thousand patterns over 128 MiB. /usr/bin/grep 13 minutes 56 seconds, ripgrep 0.171 seconds

If grep -F -f patterns.txt has been running for hours on a Mac, the size of your file probably isn't the problem. The grep in /usr/bin is BSD grep, and it runs every pattern over every line, so what it costs you is your pattern count multiplied by the size of your file. Two things to run instead:

# ripgrep: matches anywhere on the line, the way the grep you have does.
brew install ripgrep
pigz -dc data.csv.gz \
  | { IFS= read -r h; printf '%s\n' "$h"; rg -aF -f ids.txt; } \
  | pigz -p 8 > filtered.csv.gz

# mawk: faster still, when the match sits at a known position in the line.
pigz -dc data.csv.gz \
  | LC_ALL=C mawk 'NR==FNR{p[$0];next} FNR==1||(substr($0,1,36) in p)' ids.txt - \
  | pigz -p 8 > filtered.csv.gz

I ran into this filtering 1122 universally unique identifiers (UUIDs), one per line in ids.txt, out of a comma-separated values (CSV) export of 330 GB (329,691,473,398 bytes) and 1.27 billion (1,272,998,467) lines, 10.3 GB gzipped. BSD grep was heading for a month. The mawk pipeline took 3 minutes 40 seconds.

The Cost Is 6.5 Milliseconds per Pattern per Mebibyte

Whether this will bite you comes down to two numbers, how many patterns you have and how big the file is. I timed three pattern counts against three slice sizes of that file, twice each, on an Apple M5 running macOS 26.5.2 with BSD grep 2.6.0-FreeBSD:

Patterns	8 MiB	32 MiB	128 MiB
10	0.58 s	2.14 s	8.70 s
100	5.12 s	20.62 s	82.20 s
1000	50.47 s	206.44 s	835.69 s

Every cell is the mean of two runs, which were within a 5% spread at the smallest and a 0.5% spread at the largest. Multiply the pattern count by the mebibytes and then by 6.5 milliseconds, and you have the time. Eight of the nine sit within 5% of a flat rate over a span of 1600 in the product and 1440 in the time; only the smallest, at half a second, is further out, and that's start-up.

So the two figures to keep are that about 1500 pattern-mebibytes is ten seconds, and about 9000 is a minute. A 2 GiB file against 50 patterns is 102,400 of them, which is eleven minutes. My 1122 patterns against 330 GB were 353 million, which is 26 days, and from the two progress checks below I got 27 days and 29, independently of it.

If your own product is in the low hundreds, none of this matters and BSD grep is fine. The rest of this page is for everyone above that.

Every Line Against Every Pattern Is a Nested Loop Join

BSD grep reads a line, then evaluates the pattern set against it one pattern at a time: your lines on the outside, your patterns on the inside, which is a nested loop join. The comparison count is the Cartesian product of the two, and each of those comparisons has to scan the line it lands on, which is how a count of lines turns into a count of bytes and gives you the rate in the table above. All the slices I timed came out of one file, where the lines and the bytes rise together, so these runs don't separate the two; the per-comparison scan is the reason to expect bytes to be the factor that matters.

-f is only the usual way to hand grep a large set, not the trigger. Over the same 2 MiB, a hundred patterns as a hundred -e flags cost 1.20s where -f costs 1.21s, and folding them into one -E alternation is 1.6x faster at 0.75s, against the hundredth of a second GNU grep takes on that identical alternation. If you have a lot of patterns, how you spell them barely matters.

GNU grep compiles the pattern set into one automaton instead and makes a single pass, so adding patterns to it is nearly free. mawk loads them into a hash and looks up one substring per line, which arrives at the same place by a different route.

The same thing happens in DuckDB, where IN against a list of 1122 identifiers compiles to a hash join. Build one lookup structure and a row costs a single lookup however large the set is. Loop over the set instead and a row costs one comparison per entry, so the work is rows times entries, which is the multiplication in the table above.

The timings separate the two greps the same way. Over the same 128 MiB with a thousand patterns, BSD grep takes 13m56s where GNU grep takes 0.401s, 2084x faster, and ripgrep 0.171s, the 4887x on the image at the top. Their scaling separates them too: on that slice a hundred times the patterns costs BSD grep 96 times the time and GNU grep 3.2 times. mawk takes 0.045s whatever the pattern count.

Reach for ripgrep, Unless the Match Sits at a Known Position

Here are the three that will do this job at a usable speed, on the whole 330 GB with all 1122 patterns, decompressing with pigz and writing the same gzipped output through pigz -p 8, with the floor and the macOS default for scale:

Wall time for one filter over the full 330 GB. Decompressing and counting the bytes, the floor, 1m44s at 3164 MB/s. mawk 3m40s at 1500 MB/s. ripgrep 7m05s at 776 MB/s. GNU grep 17m10s at 320 MB/s. /usr/bin/grep killed at 15 hours, 0.13 MB/s, having read 2.16% of the input, too slow to fit the same axis.

The top bar is the floor, the time to decompress 330 GB and count the bytes, because everything under it has to do that too. All three matchers selected the same rows in the same order, so they differ only in how long they take: GNU grep's output and ripgrep's are byte-identical, and mawk's is those same bytes with one extra line at the top. Every rate here is this machine, this file, and this one month of data. Compare them with each other rather than with numbers from yours.

All three cost one brew install, so the choice is not about effort:

ripgrep, the one to reach for. brew install ripgrep, and it matches anywhere on the line the way grep does, so nothing about your patterns has to change. It carries an Aho-Corasick automaton, reads each input byte once however many patterns you hand it, and is 2.4x faster than GNU grep here. Pass -a: without it a stray byte sequence makes ripgrep decide the stream is binary and stop early, and a file that is quietly short is a worse outcome than a run that is slow.
mawk with a hash, if the match sits at a known position. Fastest of the three and the most restricted. The pipeline at the top loads the identifiers into the array p and tests substr($0,1,36) against it, so it matches field 1 exactly rather than the identifier anywhere on the line. That's stricter than grep and it was what I actually wanted; if your patterns can appear anywhere, this one doesn't apply to you. If they can't, you can say so by anchoring every pattern to ^uuid,. Anchoring is worth 1.06x to ripgrep and 2.31x to GNU grep, each against its own unanchored run, at this file's 12.6% match rate, and the GNU grep figure rises to 6.38x at a 1.5% match rate, because the sparser the matches, the bigger the gain; ripgrep stays between 1.06x and 1.08x throughout. mawk still finishes first, by 1.83x over anchored ripgrep and 2.04x over anchored GNU grep.
GNU grep, if you'd rather not learn another tool. brew install grep puts it at ggrep, the flags are the ones you're already typing, and your pipeline changes by one character. That costs you 2.4x against ripgrep, or 1.11x at this file's match rate if both are anchored. Where matches are sparser the anchored order flips: at a 1.5% match rate anchored GNU grep takes 2m54s against anchored ripgrep's 6m56s. Either way it finishes a job the default could not.

The greps drop your header row. A CSV header matches no pattern, so grep and rg throw it away, and that is the extra line mawk wrote: 83 bytes, twelve of them once gzipped. Both commands at the top put it back, by different means. mawk has a record number, so FNR==1|| prints line 1 whatever is on it. grep and ripgrep have no such notion, so the shell peels the line off the front of the stream instead: read takes exactly one line from the shared stdin and the filter picks up from line 2, with IFS= to keep the header's leading and trailing whitespace and -r to stop backslashes being read as escapes.

Three traps, all of them silent, and two are in ids.txt rather than in your data. One empty line in the pattern file matches every input line, so the filter quietly becomes cat: 7,794 matching lines in a 200 MiB sample became all 784,857. The tell runs the wrong way, because an empty pattern matches before any real work happens and the run gets faster, BSD grep going from 10.65s to 0.01s on a 2 MiB slice. A job that has been crawling and then suddenly finishes is the thing to be suspicious of. More empty lines change nothing, and neither does where they sit, so one stray newline at the end of ids.txt will do it.

CRLF line endings in the same file do the reverse. grep appends the carriage return to every pattern, so nothing matches and you get an empty file and exit status 1. That costs more than the real job, 14.30s against 10.65s on the same slice, because a line that matches lets grep stop trying patterns on it and a line that cannot match never does. ripgrep strips the carriage return and is unaffected. mawk is immune to the blank line, an empty string never equalling a 36-byte prefix, but not to the CRLF, where it writes your header and nothing else.

The third trap is in the data. All three matchers read a line at a time, so a newline inside a quoted field, which the CSV spec allows, splits one record across two lines and defeats them equally: the second line starts mid-record, and a byte offset lands on whatever happens to be there. This file has none, which is why substr($0,1,36) is safe on it. DuckDB's CSV reader counted 1,272,998,467 records where awk counted 1,272,998,467 lines, and two counts that match, one of them from a real parser, are how you know.

The Same Command Name Is a Different Program

This goes past grep, and it's the part that holds at any size: the name of a command tells you almost nothing about which program will run, and you can't reason about its speed until you know. sed, sort, date, find, tar, head -c and tr all differ between macOS and Linux, in flags and in performance, and inside an Alpine container most of them are busybox, which is a third implementation again. grep --version costs nothing and I should have run it first.

Fifteen Hours In, grep Had Written 2.7% of the Output

I started with pigz -dc | grep -F -f | pigz -p 8, which is what I'd typically use, and went to do something else. Four and a half hours later the output file was 11 MB and at fifteen hours 38 MB. I had no denominator for either until ripgrep finished the same filter in seven minutes: 0.8% and 2.7% of the finished output. The rows in that output sit 0.70% and 2.16% into the input, which is how far grep had actually reached, at 0.13 to 0.14 MB/s. From both checks I got about a month, so I killed it. Having something to compare against is what made stopping obvious rather than agonising.

The long version has the six matchers timed against each other, the awk field-splitting measurements, what quoted fields do to awk -F,, and the benchmarks that misled me.

Review: Brief: Make a Bigger Impact by Saying Less

2018-05-30T20:51:00-07:00

Verdict: Your Mileage May Vary

The Good

There is good, solid information in the first 8 chapters of the book. It's easy to read and absorb. Applying the ideas as a habit will take some effort on your own.

The Bad

Pretty much all the stuff after chapter 8 is just fluff. It would even be too generous of a compliment to say that the remainder of the book is peripherally related to brevity.

The Ugly

The stories that go on and on and on.

Notes

Disclaimer: These are primarily written for my own future reference, but they may be useful to you, either to decide if you want to read / buy the book or as something to revisit. The information is not comprehensive, not in the least - so, please don't use it as a substitute for actually reading the book. As with all my content, some is verbatim from the original source, opinion may be interspersed & of course, YMMV.

Being brief is not just about time. What's more important is how long it feels to the audience. It's not about using the least amount of time. It's about making the most of the time you have.
It's a balancing act of being concise, clear, and compelling. All three need to be in harmony.
To be brief doesn't just mean being concise. Your responsibility is to balance how long it takes to convey a message well enough to cause a person to act on it. That's the harmony of brevity when it's striking the right chords.
Brevity starts with deep expertise. Only with thorough knowledge can you accurately make a summary.
The road to brevity hard work and lots of time. Doing all the digging and analysis on your own time saves the members of your audience from doing the labor themselves.

7 Capital Sins

Your leadership team doesn't know what to do with your assessment & guidance & subconsciously starts questioning your leadership ability... When someone who is braver & clearer steps up, you're history.
Nobody likes to talk to you because once you start, there is little they can do to stop you... People ignore or instantly delete your e-mails. Over the course of your career, your specialty knowledge & knack for being long-winded impede your career growth.
If people see that you don't respect their time, they'll stop respecting you. During your review, you get low scores from your peers.
There's a time & place for small talk. You must always be aware that what is exciting for you to share could be agony for someone else. You should treat everyone the same way; they are busy & begging you to be brief.
Even though your idea is still developing & in lots of pieces, people will make judgments about you & your abilities. Choose carefully when & where—& to whom—you think out loud.
When people want a simple answer, you need to give it to them... (else) they lose patience & trust.
Being mentally & verbally lazy sends a clear message: you might not be ready for the next assignment—or even be the right person for the job.

Trimming: Level 1 to 3 Details

Think of a defining professional moment.
Take 10-15 minutes to prepare for a 3-minute presentation.
Present to a partner who listens, takes notes, and shares a 1-minute version to the rest of the group.
When telling a story, Level 1 details are those that are absolutely essential; Level 2 points are those that add a little flavor but do not take too much more time; Level 3 items weigh it down and don't make the story noticeably better.
This challenge forces you to start trimming. To hit the 1-minute mark means only Level 1 and a few Level 2 details.
Implications:
1. Ensure instant appreciation: People are thankful when it requires less energy to grasp the same basic information in less time.
2. Look for what weighs you down: Cutting out the finer detail requires attention and awareness for what unnecessarily burdens people.
3. Live the "less is more" way: When you are throwing things out, it may be hard to decide what goes - but keep in mind what people will really care about.

Outlines

Outlines organize and prioritize thoughts with clarity and logic, and help stop nonsensical rambling.

Five immediate benefits to outlining are they keep you:

Prepared: I'm ready to deliver.
Organized: I understand how all my ideas connect.
Clear: I'm certain what my point is.
Contextual: I can draw a bigger picture so my point stands out.
Confident: I know what to say, inside and out.

Mind Mapping

Mind mapping software is particularly well suited to information triage, because it enables you to gather, organize, evaluate, and take action on a vast amount of information, knowledge, and ideas…by giving you the power to move topics and subtopics around, attach notes, links, and documents to them that provide meaning and context, and play “what if” with your ideas. No other type of software provides the power and flexibility offered by mind mapping software for manipulating information and ideas.

BRIEF Maps

Every BRIEF Map is organized in the following way:

B: Background or beginning
R: Reason or relevance
I: Information for inclusion
E: Ending or conclusion
F: Follow-up or questions you expect to be asked or that you might ask

Controlled Conversations

Do not use your turn to start an irrelevant conversation.
Keep your responses short.
Know when to end so the other person can begin talking again.

Raise the Bar: Ennobling Habits

2018-05-23T20:34:00-07:00

I've been working at my currentjob for almost a year now and it's been a fun roller coaster ride the entire time. One of a handful of reasons I decided to work at here was that over the course of my interview, I got a strong sense that the engineers I spoke with truly enjoyed working there. When I asked them about problems they faced at their job, I got very straight answers; nothing canned or evasive like I had encountered at most other places I was interviewing with around the same time.

I recently changed roles at my current job and became officially responsible for running a team. One of the first changes I've been able to effect, with enthusiastic support from fellow engineering directors and our head of engineering, is to set up a cross-team mentoring program. Our site reliability engineering (SRE) team lead, was a universal early pick. I had an interesting conversation with her earlier today, which in a way, serves to emphasize a prized quality in high-performing engineers -- constantly raising the bar. The chat was in the context of making improvements; how a (short) period of a problem festering followed by some trigger pushed her over the line to actually force a change in behavior to eliminate that type of issue from recurring.

We function best when our habits ennoble us.

I'm paraphrasing, so apologies in advance if it lacks precision:

➜ I can only walk into a room and see a candy wrapper on the table so many times before I have to pick it up and toss it into the trash can. But it doesn't end there. I start to think about why the candy wrapper wasn't put into the trash can right away and when I look at the room, it becomes obvious. It was easy to leave the candy wrapper on the table because there was no trash can in the room. That's an easy problem to solve - put a trash can in the room, right next to the table; this makes it easy to do the right thing (put the candy wrapper in the garbage) and just that much more difficult to do the wrong thing (leave it on the table). It's not enough to make it easy to do the right thing, we have to make it hard to do the wrong thing. "We function best when our habits ennoble us." As we develop such habits, we raise the bar - a little bit at a time, each time - and it accumulates as we do it repeatedly.

➜ Software suffers from the problem where it's acceptable to have a 15-page instruction manual to do simple things and the barrier to entry for software developers wanting to contribute to opens source projects is needlessly high. We, as engineers, are part of the problem, in that we've accepted that it's okay for the barrier to entry to be that high, since we're used to dealing with complexity as part of our jobs.

➜ We need to design our software with an easy button.

Review: Kanban in Action

2018-05-14T13:50:00-07:00

Authors: Marcus Hammarberg & Joakim Sundén

Verdict: Highly Recommended

Buy the book and hold on to it as a reference.

The Good

A very solid introduction to Kanban.
Fairly in-depth on the meta-process and gradual, piece-meal incorporation into existing workflows.
Fable style introduction to Kanban in chapter 1 - I was able to recall most of the chapter from when I first read it back in June / July of 2017, a good 9 months later.
Excellent pointers to reference material and Kanban-adjacent information to continue to build one's knowledge.
Strong focus on constant improvement.
Summary section at the end of each chapter provides a quick refresher.

The Bad

The chapter on pitfalls seems to pander to critics.
The last chapter "Kanban Games" might have been better introduced earlier; in retrospect, the "Passing the Pennies" game was a good starting point to explain WIP limits and need to focus on lead times.

The Ugly

Nothing.

Notes

Kanban Principles

Kanban aims to make the work flow fast through the whole value chain, from idea or concept to software in production, delighting your customers.

Visualize your work and policies.
Limit the amount of work in progress.
Control flow of work.

Visualizing Your Work

Visualizing your work is ultimately about creating the transparency and information-sharing needed to understand how the work works and to collaborate effectively together.
The process of visualizing work means making information visible that previously wasn’t, making implicit knowledge and policies explicit, and, in doing so, resolving inconsistencies and conflicts that surface.
The kanban board is a great way to visualize your workflow and share information about priorities, who’s working on what, the progress of individual work items, and so on.
By using a big, visible board, the information radiates to everyone interested instead of being hidden in people’s brains or inaccessible tools.
Creating a kanban board is easy: you map the workflow of your work items - that is, the steps they typically go through in order to be completed—to columns you create on a whiteboard or a wall.

Work Items

A work item should contain all the information the team needs to be able to know how to work with it. Of the following, pick and choose what fits your current needs and add features as needed:

A description - So you know what the work is about
An avatar or other marker - So you know who is working on the item
Deadlines and other important dates - So you know when you need to have the item done
Tracking IDs or other references to an external system - So you know where to find more information
Blockers - So you can pick out items that are blocked and therefore hindered from further progress
Type of work - So you know the type of work for each item, which is important so you can prioritize the work items against each other if needed
Progress indicators - So you know how much of the work you have done so far
Size - So you can see the differences in size and effort
Flow data, emotions, and other data accumulated during the flow through the board - So you know how the work behaved and how the team felt about it

Work in Process (WIP)

The more WIP, the longer the cycle time for each work item will be. You should limit the WIP to get faster flow and shorter lead times.
WIP can take many forms, and we looked at some common ones in software development:
- Specifications that haven’t been implemented yet
- Non-integrated code that “works on my computer”
- Untested code, which may or may not live up to your standards
- Code not in production, which sits and wait for the next release
Problems and bad things can follow from having a lot of WIP:
- More context switching
- Prolonged feedback loop, which in turn causes extra work for you
And we mentioned some ways that WIP can manifest itself:
- Increased risk
- More overhead
- Lower quality
- Less motivation

Limiting WIP

There’s no one right WIP limit for you and your team.
Limiting WIP isn’t the goal; improving flow is. WIP limits are merely a tool that helps find problems that hinder a better flow.
A lower WIP limit will move your work faster and also bring problems to your attention more quickly. You want a low WIP limit, but not too low.
When setting out to find a WIP limit, start simple—maybe as simple as the slogan “Stop starting, start finishing.”
You can limit the WIP for the whole team/workflow:
- This approach can help you improve collaboration.
- It helps you keep focus on finishing work items before taking on new ones.
You can limit WIP based on column:
- This puts focus on flowing items through the workflow.
- It can help to manage bottlenecks.
- Maybe you already have a hunch about what you need to improve and can start limiting WIP for that column.
- Work can also be limited by estimated size (story points, for example).
You can limit WIP based on people:
- Limiting the number of avatars that can be in play is one way to easily accomplish this.
- Using swim lanes is another.
- This can help you prevent too much multitasking and people being involved in every item on the board.
Most teams have a WIP limit on the work-item level, not the task level.

"The question is, are you selling keystrokes or completed features?"

Managing Flow

Flow (or continuous flow) is an ideal state that describes a process in which every step in the process creates value, with no interruptions or waiting time.
Pursuing this ideal state is a never-ending quest that will help you not only get a smoother, faster flow but also find problems in your process.
Everything that isn’t value in the eyes of the customer is waste. Eliminating waste leads to better flow.
“Managing flow” is one of the principles of kanban, and there are a lot of things you can do to help the work flow:
- Limit WIP.
- Reduce waiting times in your process: for example, by ensuring that the work is always ready for the next step or by making the work items smaller so that they move faster through the process. Remove blockers as fast as possible. Or why not strive to “never be blocked”?
- Avoid rework by building quality into your process from the start. How much of the demand you have in your system today is failure demand? How can you have less failure demand and more value demand?
- Create cross-functional teams that can minimize waiting times.
- Use goals and targets like SLAs to get clear timeboxes and objectives to strive toward.
Many teams use the daily standup to collaborate well together. Here are some practices to help make a standup great:
- Keep it short and energized: never more than 15 minutes (that’s why you stand up).
- Keep it regular: same time and place every day.
- Keep it disciplined: start and end on time.
- Keep it focused: complete discussions afterward.
Many kanban teams have some practices that they follow:
- Focus on the work, not the workers.
- Walk through the items on the board from right to left.
- Focus on deviations - the smells in your process.
Here are some other things to think about to get the most out of your standup:
- Is all the work visible on the board?
- Are we working on the right thing?
- Do we understand the work?
- Encourage spontaneous kaizen meetings after the standup in which small improvements are made every day.
Standups can be scaled to more than one team, but make sure each layer adds value and isn’t just another reporting instance. Think about the following:
- Run before or after smaller team standups?
- Who attends?
- What kinds of visualizations are needed?
- How will statuses be synchronized between local team boards and other boards?
"What should I be doing next?" is a common question at standups. With explicit policies on how to handle that question, you minimize the risk of hindering the flow of work.
A bottleneck is something that slows down production. With queues, you can get leading indicators and discover the bottlenecks.
The Theory of Constraints is a management theory built around finding and eliminating bottlenecks in a process.

An activity in itself isn’t a value-add or waste; it’s the return of time invested (ROTI) in that activity which determines if it’s waste or not.

Optimizing for flow vs. optimizing for resource utilization is a strategy decision. Lean as an operations strategy aims to increase resource efficiency through focusing on increasing flow efficiency.

The Seven Wastes of Software Development

Partially done work - Work that isn’t completely done is in reality just work in process. It lies around waiting for you to complete it and adds to your WIP.
Extra features - Taiichi Ohno famously said, “There’s no greater waste than overproduction.” Build the right thing. On an individual level, if you’ve ever thought “This may come in handy” or “Maybe they want this to be configurable,” it's likely wasted.
Relearning - Software development is, to a great extent, learning. Failing to remember mistakes you’ve made before and having to redo them (and relearn the solution) is a great waste. This can happen both in teams and for individuals.
Hand-offs - When you hand work from one person to another, a lot of extra work is created in order to convey necessary information to the next person. Even with that extra work, a lot of information will be lost in the hand-off.
Delays - Delays create extra work.
Task switching - Context switching and the wasteful effect it can have on your focus and capacity can be as high as 10% per switch.
Defects - Defects are work that comes back to you because you did something wrong the first time around. Not only does this create extra work, but typically that kind of work comes at a bad time, slowing down the things you’re working on right now.

Specification by Example: Write your specification in the form of acceptance criteria or concrete examples (real numbers and real data used) of how a feature will be used. The feature will be complete when the example is fulfilled: nothing more, nothing less.

Five Focusing Steps

Identify the constraint of the system: for example, a single machine that all the parts in a factory need to pass through.
Exploit the constraint to get the most out of it. For example, make sure the machine is running all the time.
Subordinate all other work to support the exploitation of the constraint. For example, make sure you don’t send faulty parts to the machine; that would be a waste of time for the constraint.
Elevate the constraint. For example, buy another machine to share the workload.
Rinse and repeat. Go over your system again, and see if the constraint is still the biggest constraint in the system.

Never be blocked: Like a good pool player who always makes sure the shot he is taking sets up the next shot he expects to take, every step a good agile developer takes enables the next step. A good agile developer never takes a step that stops his progress, or the progress of others.

—@unclebobmartin

Classes of Service

Classes of service are a powerful way to make your policies explicit around the service level for certain type of work.
Assigning a class of service to a work item can influence the work item: visualization, prioritization, impact on WIP, and workflow.
Classes of service help the team to self-organize around
- Work selection and scheduling
- Work distribution
- Making sure the work capacity is distributed as decided
Common classes include
- Urgent (or Expedite) - Prioritized over other work
- Fixed Delivery Date - Needs to be completed on or before a certain date
- Regular - Normal items, increasingly urgent, pulled FIFO-style
- Defects - Rework produced by bad quality (you want as few of these as possible)
- Intangible - No tangible business value now, but later: paying off technical debt
Use classes that suit your needs and situation, using the examples in this chapter as inspiration. There’s no one right answer for everyone.

Planning and Estimating

Find the right time to plan - not too early nor too late, just in time:
- Order points are one way to run event-driven planning.
- Planning more often involves stakeholders around you more and builds trust, but it can also be costly. You need to balance it so that you plan with the right frequency.
- Using Disneyland wait times is a way of showing expected delivery times.
Plans and estimates are often requested by others around you, extend the workflow upstream and downstream.
Estimating in exact terms is much harder than giving relative estimates, so prefer doing relative estimates:
- Story points and T-shirt sizes are two common ways of doing relative estimates.
Estimation techniques:
- A line of cards ➜ Pick any work item and then compare the next one to the first: smaller or bigger? When all are laid out, assign T-shirt sizes.
- Planning Poker (see below).
- Goldilocks ➜ Change work items so that they're just the right size for your convenience.
Cadence is the natural rhythm or heartbeat found in your process. In iteration based processes, there are natural cadences when iterations start and stop.
With kanban you can have the cadences you see fit for activities: reviewing, planning, demonstration, and retrospectives. You don’t have to tie it to the flow of your work.
Need for plans and estimates:
- Teams that use kanban see the need for detailed plans decreasing with the tightening of feedback loops.
- Customers don’t want plans and estimates; they want business capabilities.
- Estimating and making plans is useful as long as you’re uncovering new information.
- The #NoEstimates movement talks about abandoning estimates altogether and finding effective ways to work without estimates.

Planning Poker

Have a domain expert explain the requirement in question to the team, to ensure that everyone understands what it’s all about.
Perform the following estimation rounds, and only continue to the next round if you aren’t in agreement of the size of the estimate:
- Round 1 - Conduct silent voting using cards. No discussion, but cast votes. Show votes, for example on the count of three.
- Round 2 - Let people think for a minute or two before doing a new silent vote, still without discussion.
- Round 3 - Let the team members explain why they voted as they did, and then vote again.
- If you’re still not in agreement, then park that story for a separate session, and continue with the next requirement.

Process Improvement

Core to kanban:

continuous improvement
respect for people

Common practices for improvements:

A retrospective is a way for a team to look back on their process and see how they can improve it.
Root-cause analysis is a tool that helps you find the root problem so you don’t only fix the symptom.
Root-cause analysis also guides you to see why the problem is worth solving.
Kanban Kata is a continuous-improvement tool that is based on how Toyota makes improvements.
There are three katas within Kanban Kata:
- Daily Kata ➜ Including improvement work in your daily work
- Improvement Kata ➜ A formalized way to improve your process
- Coaching Kata ➜ Improving the learners
Kata ➜ follow a set way of working toward a goal, until the routine becomes second nature and making improvements becomes the way that you normally work.

Retrospective - Five Distinct Parts

Set the stage w/ the retrospective prime directive: Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.
Gather data: The team looks back (hence retrospective) at the past period and gathers data about what happened.
Generate insights: Analyze the data to determine what conclusions can be drawn from it.
Decide what to do: Find a suitable set of actions that can be completed before the next retrospective.
Close the retrospective: Good practice is to ask what people about the form of the retrospective and how it can be improved.

Metrics

In order to know if you’re improving or not, you measure your process behavior and analyze these metrics.
Metrics are like a visualization of your process’s health.
The following are common metrics that kanban teams use:
- Cycle time ➜ time taken to complete part of the process
- Lead time ➜ time taken to complete the whole process
- Throughput ➜ number of completed items in a week (or month or some period)
- Due date performance ➜ do you keep your promises?
- Quality ➜ do you deliver with quality?
- Value demand vs. Failure demand ➜ demand on a system caused by failure to do something or do something right for the customer
- Discarded ideas ➜ do you innovate enough or too much?
These are common diagrams that kanban teams use:
- Statistical process control (SPC) chart ➜ A visualization of lead and cycle times
- Cumulative flow diagram (CFD) ➜ Shows a lot of information about your process based on the number of items per stage in your process per day
Finding a good metric can be hard. Remember the following:
- You get what you measure.
- Don’t focus on a single metric - use balanced metrics.
- Use metrics that are easy to capture, or make them so.
- Prefer real data over estimates.
- Use metrics to improve - not to punish.

Principles of Theory of Variation

You should expect things to vary - they always do.
Understanding variation will tell you what to expect.
Work on the causes of variations, which are always found in the system.
Understanding variation tells you when something has happened.

Pitfalls

Kanban can end up just being a long flow of work that seems never-ending - it’s just work, work, work. Make sure that doesn’t happen to you by adding cadences for reviews, retrospectives, and planning as needed.
No timeboxing is suggested when using kanban, which can remove the constraint you need to finish stuff and make trade-offs to do so. To tackle this, add timeboxes to individual items; use deadlines/due dates or SLAs.
Kanban is great because it starts where you are: you don’t have to change a thing about how you work to get started with kanban. But sometimes you need a revolution:
- Nothing is hindering you from taking on new roles or ways of working.
- You can also control the tempo at which you improve by limiting WIP , more or less. Less WIP means more opportunities to improve.
Kanban is a meta-process—an improvement process for an already-existing process. Don’t remove things that work today merely because kanban doesn’t say anything about them.
The number-one mistake when starting with kanban is to begin without any WIP limits. This means there’s no constraint pushing you to improve.

Review: The Life-changing Magic of Tidying Up

2017-03-04T18:58:00-08:00

Author: Marie Kondo

Verdict: Recommended. But be ready to filter out some strange anecdotes and slightly whacky techniques.

I first read this book in January and decided to wait to review it till I could figure out if the methods in the book were practical. When I read the book, I was a rather skeptical. Some of the anecdotes make the author seem a few cards short of a full deck. E.g. throwing other people's stuff away without asking them, dumping unwanted stuff on a younger sibling, etc. So, considering that the average person is unlikely to act in such a (inappropriate?) way, I was wary; particularly of the technique where you're expected to touch each thing to determine if it brings you joy. I decided to let the ideas, bereft of the awkward bits, simmer for a few days before deciding to put them into practice.

About a week later, I realized that the ideas could apply, not just for physical possessions, but also for a lot of ideas and thoughts swirling in my head. I'm one of those people who has at least 1 side-project going on in addition to a serious hobby. I was often left feeling like I'd never done enough and there were lots of incomplete projects (mostly software) or things I wanted to get better at (photography, playing the saxophone, etc). Truth be told, I take on way more things than I realistically have time for. These would weigh heavily on my mind whenever I spent time broadly reflecting on how things were going and whether I'd made progress in my various endeavors.

Around the same time, in January, I'd started trying to re-organize my life (yeah, a bit radical) using the techniques outlined in Getting Results the Agile Way and there are some common threads in both books which apply broadly to better self-organization.

Eight weeks later, I re-read the book, this time prepared to filter out the awkward bits and focus on the core. Looking back at the last few weeks to see what I actually used, here's what I've learned:

Tidying up is about getting rid of things, not finding places to put (hide?) things away.
I interpreted the part about "keeping only the things that bring joy" along slightly more practical lines of "is this something I will use going forward?" This interpretation seemed a little less "fluffy" to me.
Tidying by category is an excellent idea. Bringing all things of the same category together gives a clearer big picture and makes it easier to decide what to get rid of; hint: bigger pile means you're likely to dispose off more.
From the part about dealing with mementos: I found that when applied to my mind, it helped me to eliminate wasteful brain cycles on past, incomplete side-projects and hobbies that I'm unlikely to resume. The focus has shifted firmly to my on-going and near-future side-projects .
From the technique to designate a place for each thing: I'm a "work hard, play hard" kind of person. The "play hard" part becomes difficult when I don't feel like I've accomplished the "work hard" part; not because I didn't work hard, but because of distractions from work remnants. Separating the two clearly has reduced the interruptions in my head, resulting in better focus and higher productivity.
I used to store stuff by fine-grained categories which made things more difficult to find. Broad classification for storing items works well. Fewer places to look for things, consequently things are easier to find.
"Empty your bag everyday" didn't work for me. I have disparate items that I carry in my bag. They get stored in different places and I found that I spent more time putting away and gathering things that needed to go into my bag, sometimes forgetting items.
Bulk buying is economical and unless I become severely space constrained, I plan to ignore its description as hoarding.

Tidying-up definitely has a positive effect on other parts of my life. There are fewer distractions, less visual noise, and reduced guilt from having to see unused items (in my case mostly books).

Review: Getting Results the Agile Way

2017-02-08T18:58:00-08:00

Author: J. D. Meier

Verdict: Recommended (but only chapters 1-4)

The book uses Agile principles as a basis for defining a results oriented approach to work and personal life. The meat of the book is in the first part, chapters 1-4 (about 60 pages). Parts II-III and the appendix (the remaining 180 pages) expand on the information in part I (needlessly?). Be warned, there is a lot of unnecessary repetition of the same information with added verbosity later in the book. As a result these negatives, I highly recommend only part I of the book.

I've tried several systems of self-organization to prioritize tasks and reduce "wasted" time. I found some of them vague (e.g. Just Fucking Ship) or unsustainable (e.g. What the Most Successful People do at Work). To clarify, those books have worked for people, they didn't work for me.

"Getting Results the Agile Way" has a small set of repeatable steps, is simple, lightweight and as a result doesn't get in the way. In fact, being dead simple makes it easy to start using and to continue using.

A focus on outcomes i.e. end results (not goals or tasks) forms the backbone of the system; first at a high level (annual), from which monthly, weekly and finally daily outcomes are successively derived. The suggestions is to keep only 3 outcomes on your radar, any more may result in lost focus. Tasks are the actions that you take towards achieving the outcomes. Also, I like the recommendation to pick one improvement each month to work on.

A strong bias towards action to overcome analysis-paralysis, time-boxing (limiting time, so energy levels are high), versioning results to avoid perfectionism, incremental results, varying your approach till one works, separate action items from informational items.

Some examples of improvements I was able to make:

I had a problem reading non-technology, non-fiction books. I'd read half a dozen pages, but couldn't tell you what I'd just read in the last 5 minutes; go back, rinse, repeat and still not absorb the material. I was able to overcome this by changing my approach:
- I used Tabata timers initially and subsequently the Pomodoro technique to stay focused and not let my mind wander.
- I also started writing 1-3 line summaries or key takeaways for each page as I read it. Within 10 days, I'd finished reading 4 books and was back at my old reading pace from about 7 years ago.
- I no longer need to write notes to focus, but I've started trying out the Kindle app's Text-to-Speech feature (2x speed) while I myself read the book - it works really well.
- For printed books, if I find myself losing concentration, I read aloud to myself.
I added time tracking to ensure that I:
- Spent at least 1 hour a day having fun.
- Set aside 8 hours a week in non-work socializing.
- Got 7-8 hours of sleep. Tracking showed that I watched shows at night to unwind. I canceled my Netflix membership, fell asleep faster and gained the benefits of higher energy levels and sharper focus.
- Became realistic about time spent on life essentials like chores, errands, etc
- Started batching up tasks that could be done simultaneously. e.g. iron my clothes while watching shows.
I would beat myself up about not making progress on my side-projects (come on, admit it - you have a few yourself). So, I started writing down how much time I had outside of life essentials (incl. sleep & meals), work, socializing & (minimal) fun (aka stress busters). Tracking time using a calendar color coded with broad categories (essentials, fun, work and socializing) showed me that I realistically had only 20 hours a week for my side-projects and hobbies. I've reset my expectations, altered my plan accordingly and don't feel frustrated at myself for failing to get as much done as I "ought to".

Notes (only Part I)

Foundation

10 Values.
10 Principles.
12 Practices.
No big upfront design.
Pave a path; find a way forward.

10 Values

Action over analysis-paralysis.
Approach over results: results as gauge.
Energy over (as in above) time.
Focus over quantity: focus is the force multiplier.
Good enough over perfection.
Growth mindset over fixed mindset.
Outcomes over activities: focus on results.
Maximize strengths: spend less time shoring up weaknesses.
System over ad-hoc: for basics.
Value-up over backlog burndown: "What's the next best thing to do?"

10 Principles

80/20: 80% action, 20% analysis.
Change your approach: tune as you go.
Continuous learning
Deliver incremental value.
Less is more: reduce work that's in-flight.
Factor out action from reference: reduce signal to noise ratio.
Set boundaries: both for time and energy.
Fix time, flex scope: time is a 1st class citizen.
Rhythm of results: habit becomes ritual => produces feeling of accomplishment.
Version your results: fight perfectionism.

12 Practices

Rule of 3: focus on vital things; a buffet you can go back to, don't overfill your plate.
Monday vision, daily outcome, Friday reflection.
Scannable outcomes: what's on your radar?
Daily outcomes: 3 action items for the day.
Weekly outcomes.
Strong week: do the toughest things that drain you early in the week to create a glide path.
Timebox your day: optimize & prioritize.
Triage incoming action items: Do / Queue / Schedule / Delegate.
Monthly improvement sprints: pick 1 thing to improve each month.
Growth mindset: get knocked down, get up again.
Action lists: Daily / Weekly / Queue / Scripts.
Reference collections: for non-actionable information.

Hot Spots

Things that are important i.e. big picture items and long term goals.
Clusters of important things going on in your life.
At a macro-level think in terms of work, personal & life.

Hot Spots
Life Frame	Work	Personal
Mind Body Emotions Career Financial Relationships Fun	Activities Active Projects Backlog	Activities Active Projects Backlog

Set boundaries: max / min time & energy; use Hot Spots to set these boundaries. e.g. max 50 hrs for work and min 3 hrs for body (workout).
Worst mistake is to continually throw more time at problems.
Increase value, improve efficiency & effectiveness => reduce time spent.

Activities

Recurring activities like administration, mentoring, status meetings for work; household chores, grocery shopping for personal.
Make a list for each big activity => enables you to see stuff at a glance.
3 most important results at the top.

Active Projects

1 list per project => this is your queue.

Backlog

1 list per major backlog item.

Monday Vision, Daily Outcomes, Friday Reflection

Deliver value over backlog burndown.
Rule of 3.
Frame results for the week.
Test for success: (re)define outcomes based on what success looks like.
Scenario driven results: e.g. laying on a hammock & enjoying the rose garden, rather than tasks like mowing the lawn. Gives work meaning & improves quality of life.
Incremental progress.
Have a buffer: at a buffer, you would take smaller portions, make multiple trips. Buffers help see opportunities. No buffer => only threats are visible.
Use timeboxes: spend energy effectively.

Monday Vision

Identify 3 outcomes for the week:

Use Hot Spots as input.
Focus on where you want to be at the end of the week.
Focus on outcomes, not activities / tasks.

Daily Outcomes

Identify 3 outcomes for the day:

You can always add more later.
Don't get caught up in backlog.
Choose most valuable things.
Track against the week's 3 outcomes.
Must, Should, Could are the 3 buckets for the daily list:
- Must => got to get these done.
- Should => nice to haves.
- Could => pipe dreams.

Friday Reflection

Focus on the pitch, not the scoreboard.
Evaluate what got done, or what didn't and why.
Be balanced: what needs improvement and what went well.
Get clarity on personal success patterns and what needs to change.

Review: User Story Mapping

2017-02-07T23:58:00-08:00

Author: Jeff Patton

Verdict: Recommended

Took me about 9 hours to read. It's a bit wordy and has some repetition, but I was able to look past that due to the quality of information. I possibly have a slight bias towards the techniques outlined because I've used them with some variations both at Expedia and at my own company, but I see that more as validation that the user story mapping works better than the traditional software requirements. This is particularly true for products whose form and function fluctuate because end-users themselves aren't aware of what they need (which is very different from what they think they want) and don't know how software can make their lives better (simpler, enable them to get more done, etc).

The Good

In-depth coverage of user story mapping.
Good use cases with sufficient detail which keep the reader engaged.
Covers how things can go wrong / off-track if the basics aren't at the forefront of your mind.
Good tips on how to get people out of the traditional requirements gathering and spec writing mindset.
Clear six-step process to story mapping, viz. frame the problem, map the big picture, explore, slice out strategies for release, learning and development.
Possibly a book to keep on your shelf if this is your primary job function.
References to books and publications for related topics.
I liked the analogy of using the classic Asteroids game (by Atari) to break down large user stories into smaller ones, but not all of them at once - too many small user stories becomes like the huge number of tiny asteroid chunks on-screen if you shoot at all the large asteroid.
Sage advice that you're not done once you release - there's work to determine if what you built is right, to figure out improvements and to create plans to work on the significant ones.
Stand-out message: Deliver half a baked cake, not a half-baked cake.

The Bad

Harping about building shared understanding. Yes, it central to user story mapping and it's good to hear cautionary tales and gotchas, but it gets old after the first couple of times.
Some analogies / exercises are a bit worn out, though I probably feel that way reading about them and they actually work well in practice.
The goldfish bowl technique to limit active participants in discussions doesn't work in my experience. It's better to exclude people that aren't necessary and fill them in later, regardless of how they may feel about it. And yes, making the determination about who is necessary is a judgement call.

The Ugly

Low resolution images in the Kindle version which you must zoom into (feature not available on the Kindle desktop app, btw), squint your eyes to be able to make anything out.

Review: Software in 30 Days

2017-02-05T23:50:00-08:00

Authors: Ken Schwaber, Jeff Sutherland

Verdict: YMMV (Your Mileage May Vary)

The authors state that the goal of the book is to provide an overview of agile methodology, but it only talks about Scrum. It's geared towards executives or decision makers who want more information & learn about its benefits. So, it's not particularly prescriptive or in-depth. I found it a little on the light side, but the information is well laid out with a fair number of case studies, so it doesn't get tedious.

The Good

If you're unfamiliar with agile, there's enough information here (including in the appendixes) to understand its benefits & what to reasonably expect both in the early stages of agile rollout as well as once it is implemented across the organization at large (see shortcomings in "The Bad" section below).
Can serve as a quick refresher if you haven't done agile in a while.
Reasonable amount of information without hand-waving.
Fairly quick read, took me about 6 hours.

The Bad

Since it's written by the creators of Scrum, there is a fair amount of bias & the downsides of Scrum or agile methods in general are not given any real treatment.
Waterfall is the whipping-boy & while its shortcomings are well-known, I've seen it work reasonably well in practice for core platform libraries & services - this is one weakness for agile, because the methodology doesn't allow for rumination or "simmer time" for design / architectural decisions based on which the foundation layer(s) is / are built. Foundation layers cannot keep changing; they must have firm, well-understood requirements, simply because other layers are built on top of them & any alterations to foundation layers will cause ripple effects all through the system. Yes, there are ways to mitigate some of that (e.g. via abstractions & loose coupling), but there are real world limits in terms of time & resources that dictate how much mitigation can be incorporated.

The Ugly

Making waterfall methodology the whipping boy, without acknowledging that when used with shorter milestones & regular integration, it works better for building foundation layers - design & architectural decisions require "simmer time" & in the real world, there is a pretty high cost to making changes to foundation layers.

Review: The Five Dysfunctions of a Team

2017-02-02T23:50:00-08:00

Author: Patrick Lencioni

Verdict: Recommended

This book is well worth the 5 hours I spent reading it. In my 10+ years of managing teams, I have encountered these dysfunctions & (bad?) behaviors in mid-level & senior-level management teams. So, even though the example used is of an executive team (C-level), the methods in the book to root out the behaviors applies at all levels of management. This would've been an invaluable read for me 12 years ago.

The Good

The fable style of writing is engaging without dramatics. Clean & crisp; a fast read. I quite enjoyed it.
There's a separate section "The Model" which outlines the five dysfunctions, symptoms thereof, their impact & an outline of potential solutions which are practical. So, if you don't like fables, skip directly to this section.
I've used most of the techniques suggested to overcome the dysfunctions & can confirm that they work in practice. A few were new to me & I'll have to see how well they apply.

The Bad

The fable style of writing may not appeal to some folks, in which case skip to "The Model" section for the actual meat of the information. To be honest, I liked the fable style.
The information within each dysfunction section within "The Model" section could be better structured to clearly & succinctly separate out causes, motivations behind behaviors indicative of causes, mitigation. Though there are separate sub-sections, the information still feels scattered about & repeated (repetition may be due to inter-relatedness, not sure).

The Ugly

Nothing.

Notes

Disclaimer: These are primarily written for my own future reference, but they may be useful to you, either to decide if you want to read / buy the book or as something to revisit. The information is not comprehensive, not in the least - so, please don't use it as a substitute for actually reading the book. As with all my content, some is verbatim from the original source, opinion may be interspersed & of course, YMMV.

Pyramid of 5 Dysfunctions

Image source

1. Absence of Trust

Trust is the confidence that their peers' intentions are good, & that here is no reason to be protective around the group.

Causes

Instinctively, people compete with peers, don't want to seem weak by asking for help, are protective of their reputation & minimize / hide mistakes.

Members of Teams with an Absence of Trust

Conceal their weakness & mistakes from one another.
Hesitate to ask for help or provide constructive feedback.
Hesitate to offer help outside their own areas of responsibility.
Jump to conclusions about the intentions & aptitudes of others without attempting to clarify them.
Fail to recognize & tap into one another's skills & experience.
Waste time & energy managing their behaviors for effect.
Hold grudges.
Dread meetings & find reasons to avoid spending time together.

Members of Trusting Teams

Admit weaknesses & mistakes.
Ask for help.
Accept questions & input about their areas of responsibility.
Give one another the benefit of the doubt before arriving at a negative conclusion.
Take risks in offering feedback & assistance.
Appreciate & tap into one another's skills & experiences.
Focus time & energy on important issues, not politics.
Offer & accept apologies w/o hesitation.
Look forward to meeting & opportunities to work as a group.

Overcoming Absence of Trust

Requires shared experience over time, multiple instances of follow-through (build credibility), & an in-depth understanding of what each team member brings to the table.
Personal histories exercise: Low risk, share information about non-sensitive information like number of siblings, hometown, unique challenges of childhood, favorite hobbies, first job & worst job. Encourages greater empath & discourages inaccurate behavioral attributions.
Team effectiveness exercise: Team members need to identify sinble most important contribution that each peer makes to the team and one area that must be improved / eliminated for the good of the team.
Personality & behavioral preference profiles: e.g. Myers-Briggs Type Indicator (MBTI), Everything DiSC - provide practical & scientifically valid behavioral descriptions of team members. Tools require participation of a licensed consultant to avoid misuse.
360-degree feedback: Higher risk since peers needs to make specific judgments & provide constructive criticism. Must be divorced entirely from compensation & formal performance evaluation to avoid political undertones, & fear impact on compensation & on repercussions.
Experiential team exercises: Not sure what these are. Need to investigate.
Revisit individual developmental areas to ensure momentum is sustained - atrophy can lead to erosion of trust.
Leader must demonstrate genuine vulnerability first to encourage building of trust.
Leader must create an environment that does not punish vulnerability - no chastisement from anyone for admission of weakness / failure, discourages trust, however subtly.

Connection to Fear of Conflict

Fear of punishment due to misinterpretation of what is said as destructive / critical when engaged in a passionate debate.

2. Fear of Conflict

Healthy long-term relationships require productive conflict, especially in business.

Causes

Desire to avoid hurt feelings.
Misguided thinking that avoiding conflict increases efficiency.

Productive ideological conflict is very different from destructive fighting & interpersonal politics. Ideological conflict is about concepts & ideas, not personality-focused, mean-spirited attacks.

Teams that Fear Conflict

Have boring meetings (? don't get this point).
Create environments where back-channel politics & personal attacks thrive.
Ignore controversial topics that are critical to team success.
Fail to tap into all the opinions & perspectives of team members.
Waste time & energy with posturing & interpersonal risk management.

Teams that Engage in Conflict

Have lively, interesting meetings (? again, don't get this point).
Extract & exploit the ideas of all team members.
Solve real problems quickly.
Minimize politics.
Put critical topics on the table for discussion.

Overcoming Fear of Conflict

Mining: Members of teams which tend to avoid conflict need to extract buried agreements, call out sensitive issues & force the team to work through them - requires objectivity & commitment to stay with the conflict till resolved.
Real-time permission: coach one another not to retreat from healthy debate. Remind people when they become uncomfortable with level of discord that it's necessary - need to be careful not to come across as patronizing.
Other tools: Thomas-Kilmann Conflict Mode Instrument, commonly know as TKI - Need to investigate, no idea about this tool.
Leader should forego the desire to protect members from harm - avoid prematurely interrupting disagreements which prevents team members from developing conflict management skills.
Leader should demonstrate restraint & allow resolution to occur naturally - can be messy.
Leader should actively take on conflict when necessary & productive.

Connection to Lack of Commitment

How can a team buy into a decision & confidently commit to anything until it has engaged in productive conflict & members' perspectives & opinions have been considered?

3. Lack of Commitment

Commitment = clarity + buy-in

Causes

Desire for consensus.
Need for certainty.

Dysfunctional teams try to hedge bets & delay important decisions till they're certain of correctness. Analysis-paralysis breeds lack of confidence. Primary reason for dangerous ripple effects on subordinates - clashes will occur when employees reporting to different execs are marching to different tunes.

A Team that Fails to Commit

Creates ambiguity among the team about direction & priorities.
Watches windows of opportunity close due to excessive analysis & unnecessary delay
Breeds lack of confidence & fear of failure
Revisits discussions & decisions again & again
Encourages second-guessing among team members

A Team that Commits

Creates clarity around direction & priorities.
Aligns the entire team around common objectives.
Develops an ability to learn from mistakes.
Takes advantage of opportunities before competitors do.
Moves forward without hesitation.
Changes direction without hesitation or guilt.

Overcoming Lack of Commitment

Cascading messaging: Explicitly review key decisions & agree to what to communicate to employees / other constituents.
Deadlines: Honor deadlines with discipline & rigidity. Also applies to milestones, since that's how misalignment is identified & addressed before costs become too great.
Contingency & worst-case scenario analysis: Reduces fear that costs of incorrect decision are survivable.
Low-risk exposure therapy: Teams with dysfunctions tend to overvalue research & analysis. Use low-risk situations to demonstrate success of decisiveness once they have had substantial discussion with sufficient information.
Leader makes the decision in case of impasse - a decision is better than no decision. Better to be bold & wrong, & to change direction just as boldly. No waffling!
Leader needs to push for closure on issues & must enforce adherence to schedules.

Connection to Avoidance of Accountability

How can someone be held accountable for something that was not made clear in the first place?

4. Avoidance of Accountability

Willingness of team members to call out one another for performance or behaviors that might hurt the team.

Causes

Members' interpersonal discomfort.
Tendency to avoid difficult conversations.

A Team that Avoids Accountability

Creates resentment among team members who have different standards of performance.
Encourages mediocrity.
Misses deadlines & key deliverables.
Places an undue burden on the team leader as the sole source of discipline.

A Team that Holds One Another Accountable

Ensures that poor performers feel pressure to improve.
Identifies potential problems quickly by questioning one another's approaches without hesitation.
Establishes respect among team members who are held to the same high standards.
Avoids excessive bureaucracy around performance management & corrective action.

Overcoming Avoidance of Accountability

Publication of goals & standards: No one can ignore agreements kept in the open.
Simple & regular progress reviews: Ensures people take action despite inclination not to.
Team rewards: Create a culture of accountability by shifting rewards to team achievement instead of individual performance. Members will pitch in when another is not pulling his / her weight.
Leader needs to be careful to not be the only source of discipline - this is indicative of members not holding one another accountable.
Leader is arbiter of discipline when team fails - this should be the exception, not the rule. Be clear that accountability is a shared team responsibility, not relegated to a consensus approach.

Connection to Inattention to Results

Absence of accountability results in team members focusing on personal advancement instead of collective results.

5. Inattention to Results

Team members care about things other than collective group goals.

Causes

Lack of specific objectives.
Lack of focus on specific objectives.

Results are not just profit, revenue or shareholder returns. Executives' goals (destination, strategy) & objectives (route, tactics) are representative of team results - these ultimately drive profit. Focus shifts to teams status & individual status when focus is not on outcome-based. Also, there is no cure for a lack of desire to win.

A Team that is not Focused on Results

Stagnates / fails to grow.
Rarely defeats competitor.
Loses achievement-oriented employees.
Encourages team members to focus on their own careers & individual goals.
Is easily distracted.

A Team that Focuses on Collective Results

Retains achievement-oriented employees.
Minimizes individualistic behavior.
Enjoys success & suffers failure acutely (meaning they are not oblivious when failing).
Benefits from individuals who subjugate their own goals / interests for the good of the team.
Avoids distractions.

Overcoming Inattention to Results

Make results clear & only rewards behaviors & actions that contribute to those results.
Public declaration of results: Team works with strong desire to achieve published results.
Results-based rewards: Tie rewards, especially compensation, to achieving specific outcomes.
Leader must set the tone for a focus on results. A fish rots from the head down, so leader must value results.
Leader must reward & give recognition only to those who make real contributions to the achievement of group goals.

Review: Start with Why

2017-01-29T13:50:00-08:00

tl;dr version: Avoid

Watch the 18 min TED Talk instead - you'll waste considerably less time.

Author: Simon Sinek

Verdict: Avoid

What should've been a 5 page essay is a 250 page book with lots of redundant information generously smeared with Apple fanboy-ism. You might want to watch his TED Talk instead - you'll waste considerably less time.

The Good (sort of): Watch the 18 Min TED Talk instead.

Focus on 1 point: Successful companies are clear about their core purpose (other than making money); the ones that aren't or deviate from / lose sight of this core purpose fail / die.

The Bad

Is it a Circle, Bullhorn or Cone? Why is it "Golden"?
The contradictions: Apple's success comes from innovation, but TiVo which also innovated failed because <vague reasons>.
States the obvious: a company needs to have a mission which derives from the vision of its leaders - in his examples, he talks about only 1 leader per company, which is not how most of us have experienced it in the real world.

The Ugly

I can't get back the 5 hours I spent on this.
Apple worship.
Reaching quite a bit to draw parallels between Golden { Circle / Bullhorn / Cone } to well-known Golden Ratio - sorry, they are no similarities.
Playing amateur neuro-scientist linking Golden { Circle / Bullhorn / Cone } parts to human brain function - falls flat on its face.

Python Mixed Metaphors

2016-07-30T22:50:00-07:00

Python violates "The Principle of Least Astonishment" when using a mutable default value as a function parameter. Yeah, it's a bold statement. But hear me out and then decide for yourself.

Principle of Least Astonishment

If you're not familiar with it, Wikipedia as a pretty good explanation about the Principle of Least Astonishment

"if a necessary feature has a high astonishment factor, it may be necessary to redesign the feature." In general engineering design contexts, the principle may be taken to mean that a component of a system should behave in a manner consistent with how users of that component are likely to expect it to behave, i.e., users should not be astonished at the way it behaves.

... the principle aims to exploit users' pre-existing knowledge to minimize the learning curve, for instance by designing interfaces that borrow heavily from "functionally similar or analogous programs with which your users are likely to be familiar." User expectations in this respect may be closely related to a particular computing platform or tradition.

The Violation

In Python, when you write this code:

my_list = []

it allocates an instance of list and assigns it to the my_list variable each time that line of code is executed.

Essentially, [] is shorthand for list() and anytime you see [] in code, you would expect that a new list object would be created. And it is perfectly fair to have such an expectation.

But that's not the behavior when you use [] as the default value for function parameter.

def list_append(value, my_list=[]):
    my_list.append(value)
    return my_list

So, if you wrote the following code to invoke list_append:

first_name_list = list_append('Vijay')
print(first_name_list)
last_name_list = list_append('Varadan')
print(last_name_list)

the output you'd see is:

['Vijay']
['Vijay', 'Varadan']

and not:

['Vijay']
['Varadan']

This result would definitely surprise me. I would expect to see 2 distinct lists with Vijay in first_name_list and Varadan in last_name_list.

Note that the example I used would yield even more confusing results, when you invoke it initially with a pre-allocated list instance, but subsequently with just one parameter.

first_names = ['Jeff']
print(list_append('Vijay', first_names)) # output is ['Jeff', Vijay']

last_names = list_append('Lu')
print(list_append('Varadan', last_names)) # output is ['Lu', 'Varadan']

phone_numbers = list_append('+1-206-548-6565')
print(phone_numbers) # output is ['Lu', 'Varadan', '+1-206-548-6565']

As you can see, this would be very, very confusing; especially since the 2nd call to list_append worked just fine and created a new list when you didn't pass one in. The 3rd invocation doesn't create a new list and append it to the new list; instead it simply appends it to the list that was created when the function was first invoked with one parameter, using the default value for the second parameter.

Explanation of the Behavior

When you come across the code fragments above, you'd expect that Python would allocate a new instance of my_list each time you called the list_append function with just one parameter. And your expectations would be dashed on the rocks of Python's inconsistency.

Why? Because Python only allocates a new list instance and assigns it to my_list the first time list_append is invoked. Subsequent invocations use the same my_list instance that was allocated the first time.

This behavior is explained best by Jim Dennis' comment for the answer about "mutable default arguments", which I'll quote here:

As the def statement is executed its arguments are evaluated by the interpreter. This creates (or rebinds) a name to a code object (the suite of the function). However, the default arguments are instantiated as objects at the time of definition. This is true of any time of defaulted object, but only significant (exposing visible semantics) when the object is mutable. There's no way of re-binding that default argument name in the function's closure although it can obviously be over-ridden for any call or the whole function can be re-defined)

To clarify further, using Robert Rossney's comment as reference, default arguments to a function are stored in a tuple. This tuple is an attribute of the function. Tuples are immutable, so the mutable once allocated, cannot be allocated again. So, the same list instance is re-used in all subsequent function invocations.

The Fix

In order to get the right behavior, you would have to set the default value of my_list to a sentinel value, check for the sentinel value in body of the function and if the check succeeded, allocate a new list and assign it to my_list. The code would look like this:

def list_append_fixed(value, my_list=None):
    # note that you must explicitly compare against None and
    # can't check for "not my_list" in the if clause because the
    # "not my_list" check will evaluate to True when an empty list
    # is passed in, which does not equate to the sentinel check
    if my_list is None:
        my_list = []
    my_list.append(value)
    return my_list

Some Thoughts

The new code fragment defining list_append_fixed feels unnatural.

The behavior for allocation in case of default parameters for functions should be the same as for any other code that is executed and should not appear as though it's special-cased for first time function execution versus subsequent executions of the same function.

Additional Thoughts (rant?)

There exists a Python idiom to check that a variable is not None; see PEP-8 and yes, it applies to default parameter values. But when checking collections (like lists or dictionaries), the recommended idiom, which covers both the None value as well as the empty collection cases, is:

if not my_list:
    # ...
    # ...

The recommended idiom for checking a collection can't be used in this scenario. Why? Well, if an empty list is passed in, the if not my_list check would pass and a new list would be needlessly allocated. The only correct check is to make an explicit check against None.

It feels like Python mixed up its metaphors.

Other References

This answer to a Stack Overflow question about "Hidden Features of Python"
There's even an anti-pattern for this exact behavior.
This behavior seems to bite enough number of developers that it's part of the Python programming FAQ

HOWTO: Develop a Prototyping Mindset

2016-03-20T23:22:00-07:00

A prototyping mindset is something that needs to be actively developed till it becomes ingrained. Knowing and applying prototyping techniques is an important part of the foundation, but that alone is not enough. So, here are some thoughts about how to develop a prototyping mindset.

I'm an engineer by training. Having worked on core systems most of my career, I have a strong tendency to choose the "right" solution over the "right now" solution almost every single time. This is a great trait for a software architect to have, since by definition, the main goal of architecture is the long term health of the systems being built. So, the trade-off is to choose the solution that is healthier for the long term rather than allow hacks in design or code. In reality, shortcuts are permitted, usually to respond to emergencies or for business / competitive reasons. But these hacks are recognized as short term fixes which are scheduled for replacement with more appropriate and viable long term solutions.

This worked out very well for me when I ran the architecture team at Hotwire in one of my previous avatars, and there was naturally, a fair amount of tension between choosing designs and solutions that met the needs of the business but didn't damage the health of the product in the long term. This push-pull or tension is a good thing as long as the culture of the company is one that respects that compromise is necessary and understands that concrete mitigation must be put in place to replace hacks / shortcuts in a timely fashion. The engineering team at Hotwire was simply superb at this. They had a "maintenance queue" to which engineers were assigned on monthly rotation. The architecture manager (me) owned this maintenance queue. It was my responsibility to prioritize maintenance items and to devise a path to get those fixes into production.

The very trait of almost always picking right solutions over "right now" solutions proved to be a major handicap when it came to running my own technology shop, but I learned a very important lesson in the process: A prototyping mindset is something that needs to be actively developed till it becomes ingrained. Knowing and applying prototyping techniques is an important part of the foundation, but that alone is not enough.

So, here are some thoughts about how to develop a prototyping mindset.

Think: Design Experiment

A prototype is a design experiment, which can broadly be classified based on their intent as functional (explore an activity), visual (appearance) and user experience. Proof of concepts are slightly more complex design experiments that typically explore two or all three of the above aspects, but the same principles apply.

Essentially, the goal is to learn something from the experiment, i.e. it answers some question, which dovetails nicely into the next part.

Think: Answer One Specific Question

Yes, only one question; and it must be very specific. A question of quantitative nature is easier to answer than a qualitative one. A qualitative question would take some additional work post-creation of the prototype and is subject to biases. An example of a specific question in game development would be: "Does changing the variability of player attack dice rolls from 2d6 to 3d4 increase or reduce boss battle times?" This question is very specific. To answer the question, one parameter needs to be changed in the game, a number of (automated) tests need to be run to measure the variance in boss battle time and an analysis of the results needs to be performed.

It may seem a bit counter-intuitive initially, but if you find that yourself seeking multiple answers from a single prototype, then you need to break down the prototype into multiple phases till each phase answers exactly one specific question. Answer the question in each phase and then combine the information to answer the larger question. You may need to introduce additional phases to accommodate for effect of multiple changes on one another, but that does not alter the fundamental premise of answering one specific question at a time.

Think: Cheap and Fast

Prototyping needs to be cheap and fast. Not just cheap or fast, but both. You have to pick the right tools for the job and you must limit the scope of what you are prototyping. The way that I was able to get a handle on this aspect was to ask, "What question does this prototype answer?" This is a very different question from "What problem does the prototype solve?", which is the absolutely wrong question to ask. Prototypes don't solve problems. They answer questions about viability of a proposed solution.

Think: Tactics, not Strategy

Strategy is about doing the right thing. Tactics are about doing things correctly. Strategy is future oriented. Tactics are in the here and now. The strategy portion was (note: past tense) making the decision to create prototypes. The tactical portion is the correct utilization of the appropriate resources (tools, people, time, etc) to validate hypotheses. Strategy is over-arching and long term. Tactics are scope-limited and short term. Strategy produces goals, road maps and measurements to ensure that things stay on course. Tactics, in our case, quickly and efficiently validate hypotheses.

Think: Good Enough, not Perfect

Prototypes are not the final product. They exist to validate (or invalidate) a solution you've come up with. They need to be good enough for that purpose. Do not waste time in spit and polish.

Think: Temporary, not Permanent

Prototypes are throwaway work. Once their purpose is fulfilled, they will probably never be seen / used again. So, limit the resources (time, money, effort) you put into them to what is needed to fulfill the goal of the prototype. No more, no less. Do not make the classic mistake of building your product on your prototype -- this is a well-trodden path to failure. You will most likely spend more time on the prototype thinking you can re-use parts of it for production and later find that the halfhearted compromises you made hobble the final product.

Think: Risk Reduction

Prototypes reduce the risk of working on some feature or idea for an extended period of time, expending large sums of money, putting in a lot of effort, only to find out that it fails to meet targets and expectations. Have a clear goal for what you expect to get out of the prototype. What risk does it mitigate? What unknown dark corner does it shine a light on?

Think: Use What I Know and Have

Try to use existing skills, tools, knowledge and other resources you have at your disposal. Do not conflate prototyping with learning a new skill or tool. It is best to spend whatever time is needed to acquire the skill, be it a new piece of software, a programming / scripting language or a project management methodology such as Scrum before you begin work on a prototype.

There's no such thing as the perfect way to prototype. There are ways that work and ways that don't.

As an example, I was told by developers at one company I worked at that Java was way faster for prototyping and iterating than C++. I proved them wrong multiple times, simply because I was considerably more familiar with C++ than I was (or they were) with Java. On that note, I've come to prefer Python (batteries included) for prototyping over these last few years. It was a skill I picked up explicitly for speeding up the iterative cycles when prototyping games. I also use it for production code these days as do companies like Google and Dropbox.

Think: I will be Wrong, Mostly

Any reasonably intelligent person already knows that s/he will be wrong more than right in unfamiliar situations. Prototyping is all about the world of the unknown and unfamiliar. So, don't forget this. It's perfectly fine to have your ideas and theories disproved many times over. Ultimately, you'll find one that is proven right. That's all you need. The cost of learning by doing via prototypes is orders of magnitude faster and cheaper than guessing wrong. You don't want to stare failure in the face after the fact.

Think: Idea Validation, not Problem Solving

Prototypes validate (or invalidate) ideas and theories. They don't solve problems. That's up to you. If done right, you already have a slew of potential solutions to the problem at hand. You build prototypes to quickly try out those solutions to determine which one(s), if any, actually will work for you in the real world.

Think: Practice in Private, Perform in Public

A few years ago, I started to learn how to play the saxophone. The early months are best described in this tweet:

Essentially, when you're start learning to play a musical instrument, no one, even your own mother, wants to hear you practice. So, you're free to make all sorts of mistakes and sound as off-key and abysmal as you please. Prototyping is a bit like that. A lot of your prototypes will simply not work. You won't put them on display for the world even to seek feedback. You'll mostly likely destroy them. And that's just part of the process.

After practicing your musical skill for some time, you will try to show off you newfangled abilities. Showing off your prototype is a bit like that. Reception to it could vary wildly. But the feedback you get will be invaluable in making changes which in turn will lead to improvements in your final product.

Your final product is the equivalent of a public performance - you want it to be stellar. Your prototypes are all those private practice sessions and rehearsals with your close friends and family - prototyping __is__ the journey, brickbats and bouquets included, of making your product successful.

Think: Feedback, not Criticism

You've figured out the purpose of your prototype, judiciously allocated resources and built your prototype. Some types of prototypes require you to show them to other people and solicit their opinions. It's important that you have the right attitude when you get feedback. Not all of it will be positive, nor should you expect it to be. When you do get positive feedback, accept it, analyze it and take it for what it's worth. When you get negative feedback, do the same thing. Set aside criticism that is not constructive, but do pay attention that which is constructive. Practice goes a long way in filtering noise from signal when you're hit with a barrage of opinions opposing your own or what you expect. It might help to keep in mind that it's not always easy to give negative feedback. Most people know it tends to usually be discarded and you might want to try and appreciate the effort taken by people who give your their honest opinion.

C++ Standard Library Gotcha: Include Headers Explicitly

2016-01-24T08:47:00-08:00

I was using the boost::log library component and when compiling my code, I ran into a rather odd error converting a derived shared pointer to a base class shared pointer, i.e. implicitly converting a shared_ptr<std::ofstream> to a const shared_ptr<std::ostream> &. Here's the code snippet, with the line causing the compiler error highlighed. Note that code has been simplified for clarity.

using namespace boost::log;

typedef sinks::asynchoronous_sink<
    sinks::text_ostream_backend> TextSink;

bool Initialize(const std::string &logFile)
{
    bool didSucceed = false;

    try
    {
        auto sink = boost::make_shared<TextSink>();
        auto lfstream
            = boost::make_shared<std::ofstream>(logFile);
        sink->locked_backend()->add_stream(lfstream); // <-- COMPILER ERROR HERE
        core::get()->add_sink(sink);
        BOOST_LOG(lgr_gtl) << "LogSystem initialized...";
        core::get()->flush();

        didSucceed = true;
    }
    catch (const std::exception& ex)
    {
        std::cout
            << "LogSystem::Initialize() failed."
            << std::endl;
        std::cout << "Details:" << std::endl;
        std::cout << ex.what() << std::endl;
    }
    return didSucceed;
}

The compiler, Visual Studio 2015 Update 1, spat out the error pointing to the line above marked with the comment COMPILER ERROR HERE:

error C2664: 'void boost::log::v2s_mt_nt6::sinks::basic_text_ostream_backend<char>::add_stream(const boost::shared_ptr<std::basic_ostream<char,std::char_traits<char>>> &)': cannot convert argument 1 from 'boost::shared_ptr<T>' to 'const boost::shared_ptr<std::basic_ostream<char,std::char_traits<char>>> &'
    with
    [
        T=std::ofstream
    ]
note: Reason: cannot convert from 'boost::shared_ptr<T>' to 'const boost::shared_ptr<std::basic_ostream<char,std::char_traits<char>>>'
    with
    [
        T=std::ofstream
    ]
note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called

I was flummoxed. std::ofstream is a typedef for std::basic_ofstream, which is a class derived from basic_ostream. So, there should be no reason why the error should be thrown. The conversion from non-const to const shouldn't be an issue either, though it's a red herring and would cause most folks to look at that first and maybe add a const qualifier. Adding the const qualifier won't make a jot of difference.

I looked up the definition of std::ofstream and sure enough, it looked like this:

typedef basic_ofstream<char, char_traits<char> > ofstream;

and a little further up in the same file is a forward declaration for basic_ofstream, which looks like this:

template<class _Elem, class _Traits = char_traits<_Elem> >
    class basic_ofstream;

The one odd thing was that Intellisense found this in the iosfwd header file and was a forward declaration. But I wanted the definition, which Intellisense stubbornly refused to show me. So, I tried finding all references which only yielded the earlier mentioned ofstream definition basic_ostream forward declaration. Even more odd.

The standard says std::ofstream is defined in the fstream header. So, I cracked that open to find the definition of std::basic_ofstream, which was just as I had expected:

template<class _Elem, class _Traits> class basic_ofstream
    : public basic_ostream<_Elem, _Traits>
{ // output stream associated with a C stream
...
...
...
};

Now, I didn't #include <iosfwd> explicitly anywhere. It was getting included by one of the standard C++ headers or one of the boost headers. MSDN says, of iosfwd:

Declares forward references to several template classes used throughout iostreams. All such template classes are defined in other standard headers. You include this header explicitly only when you need one of its declarations, but not its definition.

I figured, more along the lines of "Let me throw this at the wall and see if it sticks.", that I should include the fstream header explicitly. In hindsight, seeing that Intellisense could only show me the forward declaration, it should've been the first thing to try.

So, I explicitly added a #include <fstream> to my code. Voilà! The error disappeared and the code compiled just fine.

It's a tad irritating (Who am I kidding? It's outright frustrating.) when the compiler throws out an error like the one above, but the definitions and declarations seem to be present - yeah, in this case a forward declaration only, but I would expect the compiler to complain that it didn't have enough information (in this case the definition) to determine whether ofstream / basic_ofstream was derived from basic_ostream.

Lesson learned: Explicitly include headers that contain the definitions, do not rely on indirect inclusions.

PowerShell HOWTO: Pass Command Line Arguments From One Script to Another

2016-01-22T20:10:00-08:00

I encountered another PowerShell quirk when porting my build environment scripts. This one had to do with passing command-line arguments from one script to another. Essentially, I would arguments when invoking ``script1.ps1``, which do some processing and would pass those same arguments to ``script2.ps1``. But ``script2.ps1`` wasn't getting the arguments as one might expect. It took some jumping through hoops to figure it out.

Scenario

I have to invoke different sets of scripts conditionally for setting up the build environment based on which project(s) I want to build. This is related to porting my build environment scripts from bash to Powershell.

Let's say, push0.ps1 is the main script and there are three other scripts, viz. quake.push0.ps1, doom.push0.ps1 and wolf.push0.ps1. I want to invoke push0.ps1 with some arguments from the command line and have push0.ps1 invoke one of the *.push0.ps1 scripts while passing all the command line arguments that push0.ps1 was invoked with.

A bit of code would help make the example concrete: push0.ps1

# $args is the array of input arguments to any script
# Do some work here common to all projects
# like pushing code to the master repo for developer scripts, etc.

# invoke project specific scripts here
# $Env:Project will be set to quake, doom or wolf
# call project specific push0 script if it exists
# and pass all arguments to it
$scriptName = $Env:Project + ".push0.ps1"
if (Test-Path $scriptName) {
    & $scriptName $args
}

quake.push0.ps1

# do quake specific work here
$pushFolder = Join-Path $Env:QuakeBaseDir "win32"
cd $pushFolder
hg push

$winSpecificFolder = Join-Path $pushFolder $Env:QuakeWinSubDir
cd $winSpecificFolder
hg push

# now push any external sub-repos
$quakeExtBaseDir = Join-Path $Env:ExtBaseDir $QuakeExtWin32SubDir
$args | % {
    $extFolder = Join-Path $quakeExtBaseDir $_
    cd $extFolder
    hg push
}

doom.push0.ps1

# do doom specific work here
$pushFolder = Join-Path $Env:DoomBaseDir "win16"
cd $pushFolder
hg push

$win16SpecificFolder = Join-Path $pushFolder $Env:DoomWin16SubDir
cd $win16SpecificFolder
hg push

# now push assembly language library repos
$doomAsmBaseDir = Join-Path $Env:AsmLibBaseDir $doomAsmSubDir
$args | % {
    $asmFolder = Join-Path $doomAsmBaseDir $_
    cd $asmFolder
    hg push
}

wolf.push0.ps1

# do wolf specific work here
$pushFolder = Join-Path $Env:WolfBaseDir "dos"
cd $pushFolder
hg push

$dosSpecificFolder = Join-Path $pushFolder $env:WolfDosSubDir
cd $dosSpecificFolder
hg push

# now push dos assembly language library repos
$wolfAsmBaseDir = Join-Path $Env:AsmLibBaseDir $wolfAsmSubDir
$args | % {
    $asmFolder = Join-Path $wolfAsmBaseDir $_
    cd $asmFolder
    hg push
}

As you can see from the example above, the three project specific scripts push different sub-repos that are passed in as parameters.

Problem

The three project specific scripts {quake|doom|wolf}.push0.ps1, work fine as they are now, if they are individually invoked from Powershell. But if they are invoked from another Powershell script, then they fail processing the arguments passed in via $args.

The issue is that when $args is passed to push0.ps1, it's an array of parameters, but when you invoke another script from within push0.ps1, the $args array (i.e. all the parameters) is treated as a single parameter passed to the callee script. If I add a line to print out the arguments in quake.push0.ps1 like this:

quake.push0.ps1

...
...
Write-Host "Quake:- args[=" $args.Count "]: " $args
...
...

And invoke push0.ps1 passing in some arguments, then the output looks like this:

    $> .\push0.ps1 boost libssl lua
    Quake:- args[=1]: boost libssl lua

As you can see, even though all 3 parameters are passed in, they are seen as a single parameter.

Essentially it would appear, that when the callee script is invoked, all 3 parameters as passed in as $args[0]. Now, Powershell is an object scripting language and it seems to pass an array of strings from the caller script as the first and only parameter to the callee script.

Knowing that since Powershell is built on top of .NET and deals with objects, we can verify this by checking the type of $args[0] like this:

quake.push0.ps1

...
...
Write-Host $args[0].GetType().IsArray
...
...

Output

    True

And we're in business! :-)

Solution

There are 2 solutions. The first uses PowerShell splatting, available as of PowerShell V2, but not easily found unless you use the term splatting in your search. The second involves manual expansion, which I prefer even though it involves more changes. The second method can also be used on the command line when piping the output of native programs. Programs output raw strings rather than string arrays, since they don't care about the shell they're being invoked from.

Solution 1: Using Splatting

Simply invoke the callees by prefixing the args array with an @ sign rather than the $ sign, like this (see comment marked as CHANGE:):

push0.ps1

# $args is the array of input arguments to any script
# Do some work here common to all projects
# like pushing code to the master repo for developer scripts, etc.

# invoke project specific scripts here
# $Env:Project will be set to quake, doom or wolf
# call project specific push0 script if it exists
# and pass all arguments to it
$scriptName = $Env:Project + ".push0.ps1"
if (Test-Path $scriptName) {
    & $scriptName @args # CHANGE: use @args instead of $args
}

quake.push0.ps1 - no change

doom.push0.ps1 - no change

wolf.push0.ps1 - no change

Solution 2: Manual Expansion

What we need to do in the callees, is check if the incoming $args has only 1 parameter and if it's an array. If so, then simply use $args[0] as the actual list of arguments and process it below.

Since this bit of the code is common to multiple scripts, I put it in a function and stick it in the $PROFILE file or some other file that can be dot-sourced. Here's the function:

function UnwrapArguments($params) {
    $unwrapped = $params
    if ($params -and $params.Count -eq 1 -and $params[0].GetType().IsArray) {
        $unwrapped = $params[0]
    }
    return $unwrapped
}

We make 0 (zero) changes in the caller and only 2 changes per callee. At the top of each callee file, we call UnwrapArguments and store it in $myargs (marked with comment: CHANGE #1) and further down in the script, we use $myargs instead of $args (marked with comment: CHANGE #2).

So our scripts end up looking like this:

push0.ps1 - no changes.

quake.push0.ps1

$myargs = UnwrapArguments($args) # CHANGE #1

# do quake specific work here
$pushFolder = Join-Path $Env:QuakeBaseDir "win32"
cd $pushFolder
hg push

$winSpecificFolder = Join-Path $pushFolder $Env:QuakeWinSubDir
cd $winSpecificFolder
hg push

# now push any external sub-repos
$quakeExtBaseDir = Join-Path $Env:ExtBaseDir $QuakeExtWin32SubDir
$myargs</span> | % { # CHANGE #2, we use the unwrapped $myargs instead of $args
    $extFolder = Join-Path $quakeExtBaseDir $_
    cd $extFolder
    hg push
}

doom.push0.ps1

$myargs = UnwrapArguments($args) # CHANGE #1

# do doom specific work here
$pushFolder = Join-Path $Env:DoomBaseDir "win16"
cd $pushFolder
hg push

$win16SpecificFolder = Join-Path $pushFolder $Env:DoomWin16SubDir
cd $win16SpecificFolder
hg push

# now push assembly language library repos
$doomAsmBaseDir = Join-Path $Env:AsmLibBaseDir $doomAsmSubDir
$myargs</span> | % { # CHANGE #2, we use the unwrapped $myargs instead of $args
    $asmFolder = Join-Path $doomAsmBaseDir $_
    cd $asmFolder
    hg push
}

wolf.push0.ps1

$myargs = UnwrapArguments($args) # CHANGE #1

# do wolf specific work here
$pushFolder = Join-Path $Env:WolfBaseDir "dos"
cd $pushFolder
hg push

$dosSpecificFolder = Join-Path $pushFolder $env:WolfDosSubDir
cd $dosSpecificFolder
hg push

# now push dos assembly language library repos
$wolfAsmBaseDir = Join-Path $Env:AsmLibBaseDir $wolfAsmSubDir
$myargs</span> | % { # CHANGE #2, we use the unwrapped $myargs instead of $args
    $asmFolder = Join-Path $wolfAsmBaseDir $_
    cd $asmFolder
    hg push
}

That's it. We're golden.

[SOLVED] Cocos Studio: 'Cocos Framework is not installed' Error

2016-01-16T14:45:00-08:00

The Problem

Install Cocos (without the framework) and cocos2d-x separately. Cocos2d-x works fine independently - I can create C++ (as well as Lua and JavaScript) projects just fine and build the C++ projects in Visual Studio 2015. Cocos Studio, unfortunately doesn't recognize that cocos2d-x is installed and won't let me create C++ projects that I can export and use in my game written in C++.

In Cocos Studio, I get the error: Cocos Framework is not installed The Preference menu item in the Edit menu looks like this:

When I try to create a project and then add C++ as an option later, it looks like this (no engine is recognized and C++ radio option in the Project Language section is grayed out):

Easy Solution

On Windows (7 and 10), the Cocos for Win with framework 2.3.3 installation works right out of the box. So, if you want to get on with your gamedev life, just install that version and you'll be golden.

TL;DR Version

Note: I had troubles with earlier Cocos Studio versions (2.3.2.3, both with and without the framwork included) and jumped through a bunch of hoops with the non-framework installation package in order to try and get things working. Also, I realized that it's better to use the non-framework installation package and install multiple framework versions separately, in order to be able to use them (side-by-side, if needed) with Cocos Studio. So, I'm documenting (a) what needs to be done to getting things working and (b) how I figured out what needed to be done.

How to get Cocos Studio to Recognize cocos2d-x

Download and install Cocos for Windows V2.3.3 (the one without framework) to C:\Cocos
Download and unzip cocos2d-x-3.9.zip to C:\Cocos\frameworks\cocos2d-x-3.9
Download this .REG file, change the name from cocos-framework.reg_.txt to cocos-framework.reg and run it. Alternately, you can manually create the following registry entries on Windows (64 bit):
1. HKLM\Software\Wow6432Node\CocosFramework\InstallDir as a REG_SZ and set it to C:\Cocos
2. HKLM\Software\Wow6432Node\CocosFramework\3.9\EngineVersion as a REG_SZ and set it to cocos2d-x-3.9
3. HKLM\Software\Wow6432Node\CocosFramework\3.9\Path as a REG_SZ and set it to C:\Cocos\frameworks\cocos2d-x-3.9
Create a file named version with the contents cocos2d-x-3.9 in C:\Cocos\frameworks\cocos2d-x-3.9
If you also want to do Android development, then Android SDK, Android NDK, Ant and JDK paths should be set in the Platform section of the Edit/Preferences menu item, else skip this step.

The Ugly

Note that none of this is documented anywhere on the cocos2d-x site or its forums. So, I'm posting this here and will include this information and a link to this post in my answer to my own question in the cocos2d-x forum.

How I Figured it out

I used 2 computers to sort this out. I removed all previous Cocos package installations completely from both - this includes both the Cocos Studio and the cocos2d-x framework.

On my first computer (Windows 7), I installed Cocos Studio without the framework and extracted the framework to the appropriate sub-directory. Cocos Studio did not recognize that the framework was installed. I tried various suggestions that were provided in answers to identical questions in the forums. These included:

editing the C:\Cocos\Cocos\CocosStudio2\Frameworks\FrameworkVersionList.xml to ensure that the framework version I was using was listed there. It was already there. Note that this file doesn't even exist in the installation package that works out of the box. Total red herring.
editing the C:\Cocos\frameworks\cocos2d-x-3.9\tools\console\bin\cocos2d.ini file to correctly point to the cocos2d-x installation and the templates folders. They weren't, but fixing them made no difference.
removing some and then all cocos environment variables. This made no difference.
adding a version file with cocos2d-x-3.9 in the C:\Cocos\frameworks\cocos2d-x-3.9 directory. This is a necessary step, but it didn't work by itself.

I then installed the Cocos for Windows with framework v2.3.3 on my second computer (Windows 10). Then I proceeded to compare the directory layout and files and their differences. Other than the version file, I didn't find anything significant.

I figured as a last, desperate (since Cocos Studio is cross platform) step, I'd do a registry search on the (second) computer, the one that had package installed of Cocos Studio with the framework included. That yielded the three registry entries that were not present on the first computer. I added those entries to the first computer and Cocos Studio recognized that the cocos2d-x framework was installed.

Last set of steps was to revert each of the changes I'd made one by one to determine which ones were actually needed and which ones were not. It turned out that the only change from the forum answers that was actually needed in addition to the registry change was the creation of the version file (#4 in the "How to get Cocos Studio to Recognize cocos2d-x" section above).

Hope this helps anyone else looking to figure this out.

PowerShell: Dot Sourcing Scripts Quirks

2016-01-05T08:17:00-08:00

PowerShell has some rather quirky and inconsistent behavior when dot-sourcing scripts. I ran into it when porting my scripts from bash.

I spent the last couple of days of 2015 porting my build environment scripts (recommend that you read that article first) over to PowerShell. I ran into some rather quirky and inconsistent behavior when dot-sourcing scripts.

I wanted to stick to the same organization and file layout that I had for the MSYS and Linux platforms. Essentially, something like this:

~vijay
    |-$PROFILE [1]
    |-work
      |-repos
        |-0
        |-1
          |-private
            |-bin
              |-devenv.ps1 [3]
            |-developer
              |-bin
              | |-devenv.ps1 [4]
              |-project1
                |-project1.ps1 [5]
            |-artist
              |-bin
              | |-devenv.ps1
              |-project1
                |-project1.ps1
            |-vijay
              |-bin
                |-devenv.ps1 [2]

$PROFILE [1] contains the Start-Dev() function
devenv.ps1 [2] user specific overrides go here
devenv.ps1 [3] common to all projects, users and roles, like build / install directories
devenv.ps1 [4] role specific settings go here
project1.ps1 [5] project and sub-project (if any) specific settings go here

So, I started off with the 1:1 port attempt with necessary adjustments to naming conventions (camel case for variables, title case for functions, etc). So, the user profile script [1] has a Start-Dev() function which is the equivalent of startdev() in .bashrc. Just like it works for the MSYS platform, this function in turn dot-sources the user specific devenv.ps1 [2]. devenv.ps1 [2] dot-sources the common devenv.ps1 [3] and the role specific (in this case, developer) devenv.ps1 [4], which dot-sources the project specific project1.sh [5] for that role.

I expected things would work fine, but they didn't. It was a tad difficult to debug initially, since the environment variables were getting set and I could see them when I invoked the Start-Dev() function, but functions and aliases that were setup in the dot-sourced PS1 files were missing from the PowerShell environment. The inconsistency flummoxed me for a bit. So, I threw in some Write-Host statements immediately following the lines where the functions and aliases were defined and they were fine. Similarly, they existed in the caller, cascading upwards all the way to the initial dot-sourcing that happens in the Start-Dev() function.

First I tried replacing the dot-sourcing with Invoke-Expression, but as I expected based on the documentation, that didn't work either. The behavior was inconsistent - environment variables were set / updated correctly, but function and alias definitions just disappeared, poof! I tried PowerShell modules instead of scripts to see if that fixed the problem, but no dice. The solution I ended up with was to create a Start-Dev.ps1 script with the contents of the function dumped into the script's global space. I dot-sourced the script like this:

PS C:\Users\vijay> . Start-Dev 1 project1

That did the trick. All functions and aliases defined in the dot-sourced files became defined and available in the PowerShell environment. I wrapped up the porting work by adding an alias to simplify invocation of the Start-Dev.ps1 script.

Organization of Build Environment Scripts

2016-01-04T10:42:00-08:00

Consistent organization is important when building code that uses multiple external and internal libraries. It all starts with scripts that are used to setup the build environment. The scripts need to accommodate customization on per project, per role (developer, scripter, artist, etc), per user and perhaps ever per computer basis.

I use mercurial for version control. There are multiple repositories, typically 1 per game or product. If a game or product is complex enough, it will warrant multiple repos. The layout of the "sub-repos" within a project is fixed and the developer needs to ensure adherence to the layout to get the project to build. The need for multiple repos within a single project stems from having teams with different needs and schedules working on various pieces. E.g. artists and animators work on assets on a different schedule and publish content to their own repo. Developers, QA and automated build & test systems can use placeholder assets initially and subsequently integrate the final assets when they are ready. This also provides the flexibility of using a different version control system for art and sound assets which is better suited to binary files e.g. Perforce rather than mercurial.

Coming to the actual layout, it looks something like this on Linux and MSYS2:

~vijay
    |-.bashrc [1]
    |-work
      |-repos
        |-0
        |-1
          |-private
            |-bin
              |-devenv.sh [3]
            |-developer
              |-bin
              | |-devenv.sh [4]
              |-project1
                |-project1.sh [5]
            |-artist
              |-bin
              | |-devenv.sh
              |-project1
                |-project1.sh
            |-vijay
              |-bin
                |-devenv.sh [2]

.bashrc [1] contains the startdev() function
devenv.sh [2] user specific overrides go here
devenv.sh [3] common to all projects, users and roles, like build / install directories
devenv.sh [4] role specific settings go here
project1.sh [5] project and sub-project (if any) specific settings go here

The way a build environment for a project is setup is to invoke a function, say startdev(), from .bashrc [1]. This function ensures that the local repository instance (1, 2, etc) and project are set. It then dot-sources the user's devenv.sh [2] script which internally dot-sources the main devenv.sh [3] script in private/bin and the role specific (in this case developer) devenv.sh [4] script. The role specific devenv.sh [4] dot-sources the project specific script, project1.sh [5]. At every stage, the sourcing script can override settings in the sourced script. This keeps the main line settings separated out from the overrides. Similarly, role and project specific settings are separated out into their own files.

There are a couple of things that are missing from this setup. The first is the lack of per machine overrides. These would be useful if, for example, a developer worked on 2 computers side-by-side or even virtual machines; one for debug builds and another for release builds or one for server code and one for client code. The second is that all projects have settings stored in sub-folders under each role; as you can see from the example above project1 has its build environment scripts split up, under artist and developer folders.

I've chosen not to address either of the issues at this time, because the per-machine overrides are easy to add when I'll actually need that feature in the future - it's only at that time I'll know the real requirements. Essentially, YAGNI. As for the second issue, while it would remove clutter from the per-role folders and consolidate the files under each project, I haven't felt a need to change it at this time since the number of projects is realistically rather small. Once again YAGNI applies, and I can revisit and easily rejigger the organization without too much hassle.

HOWTO: Scons - Remove Extra Build Output Files

2016-01-03T20:57:00-08:00

I spent the end of 2015 and the first couple of days of 2016 porting build environment scripts to Powershell and fixing up SCons based build scripts to work with Visual Studio 2015. When doing debug builds, incremental link (.ILK) and program database (.PDB) files are generated. These are considered extra files, since they're not the actual targets (like .OBJ, .EXE, .LIB or .DLL) being built.

The recommended way to remove these files is to mark them as side effects and call the Clean() method on the environment object passing these as parameters. So, looking at this forum post, I did just that. The change to my SConscript looked something like this:

if variant == 'debug':
    myenv.SideEffect('myproject.ilk', prog)
    myenv.Clean('myproject.ilk', prog)
    ...
    ...

And lo and behold, it didn't work. Unfortunately, SCons documentation is bit bare and scattered about with information split unevenly and non-intuitively in the user guide, man page, API (object model) docs and the wiki. The wiki, after repeatedly being hacked and abused was recently moved to BitBucket, but doesn't seem to be searchable [aaaarrrrgggghhhh!]. This is even more true for information about SideEffect() and its use in conjunction with Clean(). I tried the typical sort of debugging via print statements to ensure that I wasn't munging up any values and nothing seemed amiss.

I ended up finding a clue in the wiki page for the 'SideEffect()' function - note that this page is not linked to from the main wiki page and neither is its parent. With search functionality missing for the wiki, I only found that page via a DuckDuckGo search because I was explicitly searching for documentation on the SideEffect() function. On that wiki page, I noticed the function signature which clearly showed that the order of parameters to the SideEffect() function is the reverse of that of Clean(). To be fair, the user guide does have examples and a listing with the correct parameter order - but it's not obvious or pointed out explicitly.

In my not-so-humble opinion, it is incredibly daft and very poor design to reverse parameter order in API functions. Moreover, in this case, there doesn't even seem to be a good reason to do so. It results in increased user errors, just like with me and the original forum poster who posted the initial answer.

I swapped the order of params and voilà! It worked.

if variant == 'debug':
    myenv.SideEffect(<span style="color: #ff0000;">'myproject.ilk'</span>, <span style="color: #0000ff;">prog</span>)
    myenv.Clean(<span style="color: #0000ff;">prog</span>, <span style="color: #ff0000;">'myproject.ilk'</span>)
    ...
    ...

To ensure that I didn't run into this problem elsewhere, I added a function in my SConstruct that would take care of this, with an appropriate comment.

function AddSideEffectForClean(env, side_effect, target):
    # NOTE: the order of params to SideEffect() is the reverse of Clean()
    env.SideEffect(side_effect, target)
    env.Clean(target, side_effect)

...
...
...

global_env.AddMethod(AddSideEffectForClean, "AddSideEffectForClean")

...
...
...

In my various SConscript files, I invoke it as follows:

if variant == 'debug':
    myenv.AddSideEffectForClean('myproject.ilk', target)
    myenv.AddSideEffectForClean('myproject.pdb', target)
    ...
    ...

I wasted a couple of hours on this, but it's sorted now. Hope this helps someone else looking to clean out extraneous build output files when using SCons.

Software Atrophy: An Example

2015-09-10T12:51:00-07:00

Somewhere along the way, as tends to happen with a lot of software systems, the build scripts used at Axham Games began showing signs of atrophy and the discovery led to some important lessons when it comes to engineering software.

Previously, I'd put up an article describing the component object model of my SCons based build system at Axham Games. If you haven't already read that, you might want to take a gander; it would ease understanding of what is discussed here. Somewhere along the way, as tends to happen with a lot of software systems, our build scripts began showing signs of atrophy.

Note: This is written with a view to help myself and others running development teams. This post goes into details about poorly done software development work, but the goal is not to cast blame, but to learn how to spot the signs early and make appropriate course corrections before reaching a point where a system needs to be completely re-written.

Note 2: This was originally written in Jan 2015, but was put on the back burner because I got injured in an accident. I didn't find time till now to wrap it up and publish it.

Quick Recap

In our system we treat sets of libraries as a single component and a Python class represents a component. E.g. all the boost libraries have common parameters and are group together as a single component. Each project can choose which libraries within the set to use by specifying them in the SConstruct file and overriding them within per project SConscript files.

Here's an example of the configuration used to link to Lua libraries.

env['LUA_DIR'] = util.join_path(externals_dir, build_mode, 'lua')
env['LUA_PLUGIN_LIBRARIES'] = set(['lua52'])
env['LUA_INCLUDE_DIR_EPILOG'] = 'include'
env['LUA_VARIANT'] = externals_variant
env['LUA_VERSION'] = '5-2'

We also rely on the developer having correct environment variables set for paths to tools since they can be installed in different locations on any given machine. As can be seen from the example above, we construct any paths in an OS agnostic manner using Python's standard facilities. One critical rule that has to be followed is "No hard-coded paths" since we compile using different tool chains (mingw GNU C++, Visual C++) and multiple platforms (Win32, Win64, Linux, etc).

What Went Wrong

1. Core Libraries Component

On any given platform, there are usually core libraries like user32.lib on Windows or ws2_32.lib when using mingw. Additionally, there may be libraries such as stdc++-6 which will need to be linked to when compiling with the GNU C++ compiler. One way in which some core libraries are different from others within the component are that the corresponding dynamic libs are installed either as part of the system or via an external package; an easy way to think of them is as though they are similar to the Visual C++ runtime libraries, the .NET runtime or the Java runtime.

Unfortunately, even though this had been explained to developers on the team, it was either forgotten or ignored. So, the result was that the developer set it up to copy the core dynamic libs from their location to the target directory. This is an absolute no-no for system libraries.

2. Hard-coded Path

The developer further compounded the error of copying system libraries to the target directory, by hard-coding the path of the system directory (in this case, the mingw-msys bin directory) in the core libraries component's Python class. But, it was done in a way that's not easy to detect, by constructing the path using Python's standard facilities.

dynamic_lib_path = util.join_path('c:', 'tools', 'mingw', 'bin')

Not only would this fail were mingw installed in a different directory, this is also OS specific. The build would fail in the Linux and OSX environments.

How it was Detected

Simple answer: Completely by accident. I have a couple of machines on which I do development, one of which is a laptop on which I was prototyping before I founded Axham Games. So, the various tools were installed in non-standard directories on that laptop. When I ran a build on the laptop, it promptly broke indicating that the dynamic lib in the above hard-coded path did not exist. When used properly, the build system is designed to be able to deal with tools installed in non-standard directories, since their locations are supposed to be picked up from environment variables. Hard coding the path broke it.

Why the developer did this is mystery (he is no longer at the studio); especially considering that there are pre-existing examples of how to do this according to established convention, which is to have each developer define an environment variable to specify the directory. At best, this piece of work is reflective of a poor work ethic i.e. that the developer adhered only to the letter of the rule, not its spirit. At worst, this is insidious; the hard coding of the path was "cleverly" obfuscated by using Python's standard path construction facilities and unless one looks closely, everything seems kosher.

The Fix

The correct fix is to allow any component to specify that certain libraries that are linked to do not (or should not) have dynamic libraries copied over to the target directory.

To that effect, the base Component class from which each of the individual components derive needs to be modified to store the library list of what is not to be installed. We do this by adding one member list _dynamic_libraries_dni (dni = do not install) and two properties dynamic_libraries_dni and dynamic_libraries_installable as follows:

class Component(object):
    def __init__(self):
        ...
        ...
        self._dynamic_libraries_dni = []
        ...
        ...

    ...
    ...

    @property
    def dynamic_libraries_dni(self):
        return self._dynamic_libraries_dni[:]

    @property
    def dynamic_libraries_installable(self):
         return list({ dl for dl in self._dynamic_libraries
                            if dl not in self._dynamic_libraries_dni }),

    ...
    ...

These properties and member variable propagate through to the derived classes and are used in per-project SConscript files which invoke the dynamic_libraries_installable property to determine the libraries to install.

We also remove the hard-coded path in the Core component implementation as it was incorrect and since it is no longer needed.

Quick Thoughts on Atrophy Prevention

tl;dr version: This problem could have been avoided if the build system changes had been code reviewed like all other production code.

Recalling some discussions with a former colleague, software is not a write-once-and-forget sort of thing. All software is organic in nature; it's more like a tree. It needs to be nurtured, attended to, and from time to time, pruned.

All code, particularly production code, ought to be written carefully and code-reviewed as part of the check-in process, sometimes by more than one pair of eyes. But the same level of rigor is not applied to internal support systems like build systems and developer tools. In small teams like ours, there aren't sufficient resources to pay attention at that level of detail to non-shipping code unless it is a significant pain point or as in this case, breaks in one or more environments.

Looking back at this more critically, possible only with time and distance from the series of events that preceeded this find, the crux of the issue was:

a bad hire - the hack engineer who implemented the broken solution
a manager [me] who didn't oversee the engineer's work more closely, and more importantly, terminate the engineer months prior despite sufficient warning signs. Note that the termination is not for mistakes, it's for a myriad of reasons including poor work ethic.

Object Model of My SCons-based Build System

2015-01-20T08:18:00-08:00

We used SCons as our build system at Axham Games. SCons is implemented as a Python script and a set of modules. Configuration files in SCons are also Python scripts which are executed; this makes for a very potent build system for you have all of the power of Python, its modules and extensions available for use in your build system.

In this post, I'll focus on the object model for our build system which essentially treats each group of libraries as a component. If warranted and there's sufficient interest, I'll get into details of the entire system in a future follow up post.

Primary Objectives

modularity and ease of extension; primarily be able to add libraries with minimum addition to the build system
ability to build on multiple platforms without requiring additional steps
flexibility to be able to swap library versions with minimal alterations
override build parameters at the command line

Object Model

Each set of libraries in the build system is treated as a single component. One of the major issues I ran into early on when compiling for multiple platforms and architectures, particularly when using different compiler toolchains, was that they all followed vastly different and inconsistent naming conventions. I found this to be true even for the same set of libraries built using the same compiler toolchain where only the platform differed. The most common example is the prefixing of "lib" on POSIX systems. On Windows, some libraries get built out on POSIX emulation toolchains (like mingw-gcc) with library naming conventions similar to POSIX systems, while some libraries do not.

All sets of libraries used typically have a fixed set of properties viz. include path, library path and libraries to statically or dynamically link to. So, in essence we can treat each set of libraries like boost or Ogre3d as a single component for the purposes of our build system.

Boost as an Example

Let us take boost as an example for the purposes of explaining the usage of the object model. Boost, has a rather convoluted naming convention (with good reason, probably) for its libraries, wherein the boost version is embedded in the middle of the library name with a mixture of underscores and hyphens. e.g. libboost_thread-mgw48-mt-d-1_55.dll.a

Let's break this example name down into its constituent parts:

lib - this indicates that it's a library of some sort, could be static, dynamic or a stub library you link to use a dynamic library.
boost - name of the library "group" it's a part of
thread - the actual library, in this case it's a thread library
mgw48 - compiler toolchain and version, in this case, mingw gcc 4.8
mt - multi-threaded
d - debug
1_55 - boost version
dll.a - stub library to link to use the dynamic library

This is the name of the stub multi-threaded debug thread library from boost ver 1.55 built using the mingw gcc toolchain ver 4.8. On Visual C++ 2013, this same library would be named libboost_thread-vc120-mt-gd-1_55.dll.a.

The compiler toolchain used is project specific as is the version of boost. As long as the libraries used by the project are available, the developer will be able to use the libraries by minimally specifying the versions for the toolchains used in the project.

The onerous task of building the specific library names per platform architecture and toolchain which are specific to the set of libraries is encapsulated within a simple Python class modeled as a component. All components have standard properties to provide include path(s), library path(s) and libraries to link to. These properties are assembled in the constructor of the component based on environment variables which can be overridden by the build environment and further by values specified as parameters on the command line when the build is fired off.

So, in the SConstruct file of the project (which may have multiple sub-projects), the developer specifies the following:

# externals_dir is the base directory for 3rd party libs
# plat_arch is the platform-architecture
# externals_variant is the build variant e.g. debug
env['COMPONENT_NAMES'] = set(['boost'])
env['BOOST_DIR'] = util.join_path(
      externals_dir, plat_arch, 'boost')
env['BOOST_VARIANT'] = externals_variant
env['BOOST_VERSION'] = '1-55'
env['BOOST_INCLUDE_DIR_EPILOG'] = 'include'

env['BOOST_LIBRARY_EPILOGS'] = {
      'win.mingw.x86' : {'debug'   : '-mgw48-mt-d-1_55',
                         'release' : '-mgw48-mt-1_55'},
      'win.msvc.x86' : {'debug'   : '-vc120-mt-gd-1_55',
                        'release' : '-vc120-mt-1_55'}}
env['BOOST_LIBRARIES'] = {
      'win.mingw.x86' : set(['boost_system',
                             'boost_thread',
                             'boost_program_options']),
      #Since boost::thread depends on boost_chrono in msvc alone.
      'win.msvc.x86' : set(['boost_system',
                            'boost_thread',
                            'boost_chrono',
                            'boost_program_options'])}

There are a few basic assumptions that the components of the object model makes which are documented rather than enforced via codifying rules:

the base directory for the component is specified as <COMPONENT_NAME>_DIR
libraries used by the component are specified in a nested dictionary with the platform-architecture as the outer dictionary key and the build-variant as the inner dictionary key
components may require additional parameters to be specified, which are documented

The main reason for not enforcing the rules via code checks within the build system is that we are a small team of at most four engineers and the work of codification of rules has not been seen as required. If the team grows larger (most likely indicated by the need to open a build engineer position), it may make sense codify the rules. Any errors that we run into due to missing or incorrectly specified parameters has caused the build to fail, and fail fast. Over the years, I've become a strong believer in YAGNI (you aren't going to need it) and that's one of the foundational principles of any software I write these days, including but not limited to build systems.

At this point, the developer is ready to start using and linking against boost libraries for all sub-projects, which can choose to override these values.

A Tour of C++ / Bjarne Stroustrup

2014-12-29T12:57:00-08:00

This book is great to get you started writing modern C++ code that uses most of the significant new features of C++11 and with appropriate pointers to corresponding chapters in Stroustrup's tome (which I consider an excellent reference) for in-depth study.

Verdict: Highly Recommended

Weekly Planning

2014-12-21T12:18:00-08:00

With my annual goals in mind, I then make a priority list every Sunday night of what I plan to accomplish in the next week. That priority list will include both immediate assignments and steps toward my annual goals ... I tightly schedule Monday and loosely schedule the rest of the week. Then on Monday night I schedule Tuesday more tightly, based on what’s left on the priority list and what’s come up on Monday. Tuesday night I schedule Wednesday, and so forth. I’ve usually gotten most things done by Friday, which can be a mop up day or a time for more planning.

Laura Vanderkam (What the Most Successful People Do at Work)

Habits - Anthony Trollope

2014-12-21T10:28:00-08:00

A habit, Anthony Trollope once said, “has the force of the water drop that hollows the stone. A small daily task, if it be really daily, will beat the labors of a spasmodic Hercules.”

3 Steps to New Habits

2014-12-20T06:51:00-08:00

Make it tiny.
Find a spot.
Train the cycle.

My Fingers Thank Me - Profusely!

2013-06-30T03:48:00-07:00

Grab a seat. This is some important stuff. Stuff that you normally don't pay a lot of attention to, but should. I'm talking about your health. A bit more specifically, ergonomics. Even more to the point, your finger health. As a developer, I spend the vast majority of my time writing code, which means that my primary tool for input is a keyboard. Over the three decades that I've been around computers, I've tried a large variety of keyboards and have come to the conclusion that mechanical keyboards are the only ones worth using.

When buying gear for the studio, I ordered Logitech G-105 Gaming Keyboards without doing much research. I didn't hold out much hope of getting mechanical keyboards based on the difficulties I ran into trying to get "high-end" game development workstations. A couple of weeks ago, when talking about mechanical keyboards, their lack of availability in India and the (relatively) high prices you have to pay to get your hands on one, someone mentioned that developer friends of theirs would buy mechanical keyboards on their trips to India and take them back for use overseas in the UK or USA. I was a bit surprised to hear this. Computing hardware of any kind is not the sort of thing you'd associate with being "exported" from India. You usually bring that sort of thing into the country, not take it out. Color me intrigued! The manufacturer turned out to be TVS, who most folks, including me, know for their rather high quality motor vehicles. They also have an electronics division which manufactures mechanical keyboards among other things.

I contacted my supplier and had a sample delivered. And I was floored.

The keyboard was beautiful to look at and typing on it was such pleasure. I fell in love with it instantly. It's hard to describe the immense satisfaction as you hear the click of each key as you press it down. It's like a symphony. Even typing mistakes are less irritating as pounding away on the backspace key is also pure joy. One very nice thing about the keyboard is that the spacebar key is stiffer than the others and takes more pressure to bottom out. I have a tendency to bottom out when typing due to the PC-XT keyboards I started out with in the 80s. Those keys were considerably stiffer, needed more pressure and if I recall correctly, I had to bottom out for the key press to register i.e. the actuation point was at the bottom of the switch.

So, I'm in love now. This is one of the best purchases I've made in a long, long time. I look forward to sitting down at my workstations just to hear the sweet sounds of the keyboard as I pound away on it.

A couple of things that took me about 2-3 days to get used to:

The Enter key is non-rectangular and shaped like a mirrored fat L. This design is rather uncommon these days with almost all keyboards having a rectangular Enter key with the backslash key right above that.
The backslash key is to the right of the equals key. This was the hardest to get used to. I kept hitting the Enter key for the whole of the first day when I meant to type backslash as a path separator at the command prompt. I was very nervous that I might accidentally do that while recursively removing a folder e.g. typing del /s C: when I meant to type del /s C:\junk.
The backspace key is a regular sized key rather than being double sized. This wasn't as hard to get used to.

Here's the sweet bit - the TVS Gold Mechanical Keyboards are available at about Rs. 1600 (less that USD 30 at the current exchange rates). This is very inexpensive compared to the $100 minimum you pay for other mechanical keyboards. TVS have a winner in this keyboard. Now, you can stop reading and go out and buy one (or more) for yourself and your near and dear ones - I now have 3 of them for use at both my work locations.

Verdict: Buy, definitely.

Mercurial Diffs with vim on Windows

2012-11-08T07:31:00-08:00

UPDATE 2017-Jan-17 for newer versions of DirDiff.

I had the devil of a time getting vimdiff and mercurial to play nicely with with each other on Windows for visual diffs. It worked great on posix systems using instructions from the mercurial wiki both to diff single files as well as a multiple files. But on Windows, it would only work for single files, giving me rather strange errors when trying to diff multiple files. So, I spent a bit of time getting it to work.

According to the instructions, the following changes to your mercurial.ini or .hgrc ( depending on platform ) should enable you to use vimdiff for visual diffs with mercurial. Note that you need to install the DirDiff plugin for vim.

[extensions]
hgext.extdiff=

[extdiff]
cmd.vdiff=gvim
opts.vdiff=-f '+next' '+execute "DirDiff" fnameescape(argv(0)) fnameescape(argv(1))'

If you try using the above changes as are on Windows, all you'll end up with is vim opening 7 empty buffers with names corresponding to the parts of opt.vdiff that follow the -f parameter. This happens because of differences between the way bash and cmd.exe treat single quotes and escaping. This part was relatively easy to figure out especially with the use of the --debug option to mercurial like this: hg --debug vdiff, which tells you exactly how the command line is invoked.

So what I tried next was to replace the single quotes with double quotes and escaping the double quotes surrounding DirDiff, like so:

opts.vdiff=-f "+next" "+execute "DirDiff" fnameescape(argv(0)) fnameescape(argv(1))"

That resulted in slightly more informative errors that looked like this:

Illegal file name
Error detected while processing command line:
E119: Not enough parameters for function: <SNR>12_DirDiff

Definitely an improvement since vim was correctly getting and interpreting parameters passed. This indicated I had fixed the shell escaping and any further errors were in vim land.

I spent a bit of time looking at the DirDiff plugin code - wasn't too hard, since it was just one file. But I was spinning my wheels, coz the issue occurred in the invocation of the entry point function defined within the DirDiff plugin. So, the issue was between what was being passed to vim and how vim was interpreting it.

Here's are the approximate steps ( determined empirically ) in how mercurial invokes the visual diff program specified using the extdiff plugin:

make a directory in the user's $TMP ( or %TMP% ) folder - let's call this hgdiff-base
within hgdiff-base create 2 directories - let's call these repo.original and my.changes
copy files that have changed into the my.changes folder and put unchanged, repository versions of those files into the repo.original folder
pass the two folders as arg0 and arg1 to the external diff program ( in our case vimdiff )
as soon as the external diff program exits, remove hgdiff-base and all contents

Since mercurial would cleanup the temp directories that contained the files with differences, I just fudged my tests by using existing files that as passed as parameters to vimdiff exactly as mercurial would have.

In a couple of attempts, I'd narrowed it down to the trailing backslash that was getting passed to vimdiff when multiple files were involved. Trailing slashes are fine when working with posix systems, since they use the forward slash as the path separator and not backslash like Windows does. The directory names are enclosed in double quotes when passed as arguments to vim to account for spaces in folder and file names. So, what vim sees in the case of multiple file diffs is the folder names with a trailing backslash followed immediately by a double quote. Vim ( correctly ) misinterprets the escaped double quote as part of the file or path name of the parameter passed to DirDiff. This is what resulted in the "Illegal file name" error I got earlier.

The simple solution to the problem was to remove the trailing backslash if it was present. The easiest way to do that is to use the vim built-in function substitute. This function is the same one that is invoked within the editor when you do a find-replace. So, I modified the arguments passed to vimdiff as shown below and things were hunky dory.

opts.vdiff=-f "+next" "+execute 'DirDiff' fnameescape(substitute(argv(0), '\\$', '', '')) fnameescape(substitute(argv(1), '\\$', '', ''))"

UPDATE 2017-Jan-17

If you still run into an issue after the above fix and get an error message "There is no diff at the current line!", you are most likely using a newer version of DirDiff which forces UNIX-style locale specification at the command line without checking if it's running on Windows. The fix is to add the following line to your _vimrc or .vimrc file:

let g:DirDiffForceLang=""

Mixing MSYS and native Windows vim

If you're using MinGW under MSYS and use the Windows native version of gvim.exe supplied by vim.org, then note that when launching gvim from the MSYS shell, $HOME is your MSYS home directory and that the Windows version of gvim will look for plugins in the $HOME/vimfiles/plugin - I spent a bunch of time wondering why gvim couldn't find DirDiff even though I put it in the vimfiles/plugin folder under my Windows home directory. That happens coz the MSYS shell changes your HOME directory to be your home directory in MSYS. If you were to use the vim.exe that ships with MSYS, then it looks for user plugins in $HOME/.vim/plugin where $HOME is naturally your MSYS home folder.

You might just be better off if you don't mix MSYS with native Windows vim, unless you're like me and are willing to bang your head figuring out these kinds of quirks. :-) Enjoy.

Building Ogre3D 1.8.1 on MinGW

2012-10-31T15:26:00-07:00

Ogre3D on MinGW isn't covered very well on the Ogre site or its forums. So, I made some notes about building Ogre from source on MinGW.

Assumptions: You're familiar with Windows & Unix utilities, shell scripting ( bash & cmd ), have knowledge of command line compilation and are generally fearless, like a lion.

I put all third party code in ~/external, build them to ~/build/output and install them to ~/build/install. I eschew system wide installs and prefer to use local installation where possible, but I don't fight it too much - take the case of python. I'd prefer a simple unzip of python, but that doesn't work for a bunch of stuff ( like mercurial, NumPy, pygame ) that looks for registry entries in Windows for successful installation. So, I just use the installer from the python.org site.

Building Ogre

Here's my development setup:

Platform	Windows 7 x64
Toolchain	mingw gcc on msys which you can get here.
gcc version	4.7.0 - get this with mingw
python version	2.7.3 for x86 from python.org
mercurial version	2.3.2 for x86, I use the python package. From the official site.
Ogre version	1.8.1 from the bitbucket repo
Ogre dependencies	default branch from the bitbucket repo
Boost version	1.51
DirectX SDK version	June 2010

If you run into issues, see this Stack Overflow question||

Step 1: Boost

This step is completely optional, as boost is not a required dependency. I'm planning on using multi-threading and a whole host of boost libraries. See the post Building boost on MinGW. Note: I build everything in boost, rather than just the bare minimum required for multi-threading.

Step 2: Ogre Dependencies

Get a bitbucket account if you don't already have one. Add your ssh pub key to the account - there are instructions on the bitbucket site, if you don't already know how to do this. You can use https instead of ssh, but modify the URL passed to hg clone accordingly.
Fetch the source code.

export OGRE_SOURCE_DIR=~/external/ogre/v1-8
mkdir -p $OGRE_SOURCE_DIR
cd $OGRE_SOURCE_DIR
hg clone ssh://hg@bitbucket.org/cabalistic/ogredeps
export OGREDEPS_SOURCE_DIR=$OGRE_SOURCE_DIR/ogredeps

Set up some environment vars and build the deps:

export BUILD_OUTPUT_DIR=~/build/output
export OGRE_OUTPUT_DIR=$BUILD_OUTPUT_DIR/ogre/v1-8
export OGREDEPS_OUTPUT_DIR=$OGRE_OUTPUT_DIR/ogredeps
export BUILD_INSTALL_DIR=~/build/install
export OGREDEPS_INSTALL_DIR=$BUILD_INSTALL_DIR/ogredeps
mkdir -p $OGREDEPS_OUTPUT_DIR
cd $OGREDEPS_OUTPUT_DIR

If you're using MinGW under MSYS:

cmake $OGREDEPS_SOURCE_DIR -G "MSYS Makefiles"

or if you're using MinGW without MSYS:

cmake $OGREDEPS_SOURCE_DIR -G "MinGW Makefiles"

make
mkdir $BUILD_INSTALL_DIR
make install
mv ogredeps $OGREDEPS_INSTALL_DIR

Step 3: Ogre SDK

This is rather similar to building the dependencies.

Fetch the source code.

export OGRE_SOURCE_DIR=~/external/ogre/v1-8
mkdir -p $OGRE_SOURCE_DIR
cd $OGRE_SOURCE_DIR
hg clone ssh://hg@bitbucket.org/sinbad/ogre
export OGRESDK_SOURCE_DIR=$OGRE_SOURCE_DIR/ogre

We need to tell Ogre where boost is:

export BOOST_ROOT=~/build/install/boost_1_51_0
export BOOST_INCLUDEDIR=$BOOST_ROOT/include/boost-1_51
export BOOST_LIBRARYDIR=$BOOST_ROOT/library

On posix systems, you don't need to set BOOST_INCLUDEDIR or BOOST_LIBRARYDIR, but if you do set them, BOOST_INCLUDEDIR should be set as follows: export BOOST_INCLUDEDIR=$BOOST_ROOT/include

And where the dependencies are: export OGRE_DEPENDENCIES_DIR=~/build/install/ogredeps
Set up some environment vars for build and install locations:

export BUILD_OUTPUT_DIR=~/build/output
export OGRE_OUTPUT_DIR=$BUILD_OUTPUT_DIR/ogre/v1-8
export OGRESDK_OUTPUT_DIR=$OGRE_OUTPUT_DIR/ogre
export BUILD_INSTALL_DIR=~/build/install
export OGRESDK_INSTALL_DIR=$BUILD_INSTALL_DIR/ogre
mkdir -p $OGRESDK_OUTPUT_DIR
cd $OGRESDK_OUTPUT_DIR

If you're using MinGW under MSYS:

cmake $OGRESDK_SOURCE_DIR -G "MSYS Makefiles"

orif you're using MinGW without MSYS:

cmake $OGRESDK_SOURCE_DIR -G "MinGW Makefiles"

make
mkdir $BUILD_INSTALL_DIR
make install
mv sdk $OGRESDK_INSTALL_DIR

At this point, you should have your boost stuff in #!bash ~/build/install/boost_1_51_0, ogre dependencies in #!bash ~/build/install/ogredeps and the ogre sdk in

To build applications, simply add the relevant include and library paths to the compiler and
linker options. On posix systems, add the bin directories to LD_LIBRARY_PATH environment variable.
On Windows and MinGW, add the bin directories to the PATH environment variable.

Building boost Binaries on MinGW

2012-10-19T17:14:00-07:00

I've been tied up with recruiting related work, primarily reading resumes, looking at demo reels and doing phone screens. I got a bit tired of it all a couple of days back and decided to spend a bit of time doing what I like - think of it as brief break from the Axham Games studio setup work.

Building the boost binaries is one of the optional steps for building the Ogre graphics engine from source. I ran into a few difficulties building the boost binaries on MinGW.

Assumptions: You're familiar with Windows & Unix utilities, shell scripting ( bash & cmd ), know about command line compilation and are generally fearless, like a lion.

Here's my development setup:

Platform	Windows 7 x64
Toolchain	mingw gcc on msys which you can get here.
gcc version	4.7.0 - get this with mingw
Boost version	1.51 from boost.org

Note: I build all default libraries in boost, rather than just the bare minimum required for Ogre. Takes a couple of hours on my circa 2009 Intel Core i7 laptop with 4GB RAM & Seagate Momentus XT SATA II HDDs.

Note 2: I've learned the hard way that using a tilde i.e. ~ in args that take a path to b2 under mingw, result in a directory with the name ~ being created in the current directory into which relevant files are placed. So, don't pass a ~ to b2.

Note 3: This one's slightly "duh", but readlink will fail silently if you pass it a non-existent link, file or path. So, I pre-create directories where needed.

Get boost 1.51 from boost.org.
Put it in ~/external/boost, so your boost source root will be ~/external/boost/boost_1_51_0
We're going to use this a few times in the future, so set it as an environment var.

export BOOST_SRC_ROOT=~/external/boost/boost_1_51_0
cd $BOOST_SRC_ROOT
cd tools/build/v2

To build the binary parts of boost, we first need to build the boost build system.

./bootstrap.sh --with-toolset=mingw

Note that the documentation on the boost wiki is wrong. You must pass the toolset you want to use using the above syntax for boost build to build in mingw. Simply passing gcc or mingw will cause the build to fail with something like this in the error log:

$ sh bootstrap.sh
Bootstrapping the build engine with toolset gcc...
Failed to bootstrap the build engine
Consult 'bootstrap.log' for more details
$ cat bootstrap.log
###
### Using 'gcc' toolset.
###
rm -rf bootstrap
mkdir bootstrap
gcc -o bootstrap/jam0 command.c compile.c constants.c debug.c function.c glob.c hash.c hdrmacro.c headers.c jam.c jambase.c jamgram.c lists.c make.c make1.c object.c option.c output.c parse.c pathunix.c regexp.c rules.c scan.c search.c subst.c timestamp.c variable.c modules.c strings.c filesys.c builtins.c pwd.c class.c native.c md5.c w32_getreg.c modules/set.c modules/path.c modules/regex.c modules/property-set.c modules/sequence.c modules/order.c execunix.c fileunix.c
builtins.c:33:23: fatal error: sys/wait.h: No such file or directory
compilation terminated.
execunix.c:17:26: fatal error: sys/resource.h: No such file or directory
compilation terminated.
fileunix.c:98:17: fatal error: ar.h: No such file or directory
compilation terminated.

Set output and installation directories and build.

export BUILD_OUTPUT_DIR=~/build/output
export BUILD_INSTALL_DIR=~/build/install
export BOOST_VERSION=1_51_0
export BOOST_BUILD_DIR=$BUILD_OUTPUT_DIR/boost_$BOOST_VERSION
export BOOST_INSTALL_DIR=$BUILD_INSTALL_DIR/boost_$BOOST_VERSION
mkdir -p $BOOST_BUILD_DIR
mkdir -p $BOOST_INSTALL_DIR
./b2 toolset=gcc --prefix=$BOOST_BUILD_DIR/build install
export PATH=$PATH:$BOOST_BUILD_DIR/build/bin

mingw specific: Comment out the following line ( around line 486 ) in tools/python.jam:

python-cmd = "$(python-cmd)" ; It's within this if block:

if [ version.check-jam-version 3 1 17 ] || ( [ os.name ] != NT )

Now that the boost build system is built and installed, time to build the actual boost library binaries.

cd $BOOST_SRC_ROOT
b2  --build-dir=$BOOST_BUILD_DIR toolset=gcc --prefix=$BOOST_INSTALL_DIR install 2>&1 | tee $BOOST_BUILD_DIR/build.log

That will build static and shared non-debug multi-threaded variants of the libraries. To build all variants, pass the additional option, --build-type=complete

Check the build log for errors, you should have 12 failures due to missing unicode/uversion.h which you can safely ignore ( some internal check for optional stuff that should be warnings ). At the end of the build process, you should have the boost libs installed in ~/build/output/boost_1_51_0/lib and headers in the boost subdir under ~/build/output/boost_1_51_0/include/boost-1_51.

Note: Header files in mingw are in nested one level deeper than in posix builds i.e. in posix the dir structure looks like $BOOST_ROOT/include/boost and in mingw it's $BOOST_ROOT/include/boost-1_51/boost. So, when using boost in multi-platform builds like I do, you need to adjust your include path accordingly.

Posix Build Specifics ( Ubuntu LTS 12.04 )

To get boost to build without errors under Ubuntu LTS 12.04, you need to install the following additional libraries via apt-get:

python-dev
libbz2-dev

Update: Added posix build info.

Update 2012-10-30: Fixed paths and added note to comment out line 489 in tools/python.jam

Compression is not a UI decision...

2009-05-12T19:24:00-07:00

One of the dilemmas that any software designer goes through is determining in which subsystem a particular behavior needs to be implemented and whether it should be user configurable or not. By user, I don't necessarily mean the only end user; in the case of plugins or server software, I also refer to the people installing / configuring the plugin or software.

Read on to learn how I tracked down and fixed a 10X slowdown rendering Wordpress content which was the result of a terrible design decision by a theme designer.

I'd been pondering about subsystem decomposition after I ran into horrendous performance issues with a Wordpress theme I used for my imagery blog, which uses shared hosting. A theme, for those not aware, is the UI customization that can be applied to a piece of software to change its look and feel - there's no added functionality provided i.e. they're not plugins. The theme in question is Grain - a plugin specific theme, it requires Yet Another Photo Blog. When using the specific theme for YAPB, the pages would take about 16-17 seconds to render. Since I had to put both the theme and the plugin into service at the same time, it took me some time to figure out where the problem lay.

After some head scratching, I replaced the Grain theme with another YAPB compatible theme and the pages began rendering in the standard 2-3 seconds as before. So, definitely a problem with the theme. I visited other sites, including the theme author's that used the Grain theme. They rendered pretty fast i.e. in under 2 seconds. So, something was off with my config. But what? There were a number of places things could go wrong.

Step 1: would be to use wget to retrieve a imagery blog page. Page came back in 1-2 seconds. So, it seemed it was not a problem with the page itself.

Step 2: Was it problem with the browser? Firefox is my default. So, I tried with IE. No dice. Still the same 16-17 seconds to render a page. What do browsers do that wget does not? They send headers automatically.

Great - time for step 3: look into the browser headers. I installed the Web developer plugin for Firefox. Yes, I know. Shame on me for not having it installed already. Then I got YSlow. Awesome debugging aid. Turns out it __was__ the page itself that was taking up almost all of the 16-17 seconds. Examine the headers - request and response headers look normal. The browser accepts gzipped and deflated content and the server was sending back gzipped content. Except, the compressed content was larger than the uncompressed content. Hmmm, that's odd. HTML compresses very well due to repeated patterns, so something was definitely off. The one time that content increases in size when compressed, is if it's __already__ been compressed - which is why one never compresses .jpg, .gif or .png files. Those formats are compressed internally. So, was there double compression going on? If so, how the browsers were able to double decompress is still a bit of a mystery to me, but that part was definitely working. IE doesn't have any option to turn off compression. There's the sledgehammer of disabling HTTP 1.1, but that's overkill. So, I continued w/ Firefox. You can change many, many options by typing about:config in the address bar. I disabled compressed content acceptance and viola - the imagery blog pages began rendering within 2-3 seconds.

Step 4: Time to revert the Firefox setting and start fiddling with the server. Easiest thing to do was disable compression for the imagery blog using .htaccess - my imagery blog is on a shared host using Linux / Apache. So, I proceeded to do what seemed to be the obvious, viz. add the following lines to the .htaccess file of my imagery blog:

<ifModule mod_gzip.c>
    mod_gzip_on No
</ifModule>

But, that didn't work. So, hunted for another solution, which was to use rewrite rules to disable gzip, to be added in the mod_rewrite.c section:

RewriteRule . - [E=no-gzip:1]

No, dice again. I had also added the following mod_expires section based on YSlow recommendation and those changes were getting picked up when I looked at the response headers:

<IfModule mod_expires.c>
    ExpiresActive On
    ExpiresDefault "access plus 1 seconds"
    ExpiresByType text/html "access plus 1 seconds"
    ExpiresByType image/gif "access plus 2592000 seconds"
    ExpiresByType image/jpeg "access plus 2592000 seconds"
    ExpiresByType image/png "access plus 2592000 seconds"
    ExpiresByType text/css "access plus 604800 seconds"
    ExpiresByType text/javascript "access plus 216000 seconds"
    ExpiresByType application/x-javascript "access plus 216000 seconds"
</IfModule>

Hmmm. Something else which was doing the compression was slowing things down.

Step 5: A search for php compression revealed that there was support for compression available in the standard php libraries. Bingo! Grepping for the use of php compression showed a function in the theme that enabled it if there was support available and if the (system wide) php.ini file had it enabled. Ouch! Therein lay the issue. I couldn't muck with the system wide php.ini file, since I was on a shared host. Php compression was slow, not sure why, but it was SLOW. There were no configuration options to disable php compression in the theme. So, the only way to do it was to modify the source code. I commented out a few lines to disable php compression and viola, everything worked fine thereafter.

This was quite a hassle. The major lesson here is that compression is not a UI component decision: The biggest offender here in this debacle is the Grain theme designer. Whether to use compression is not a decision for a UI component to make. Essentially, the theme designer overstepped his/her bounds. While, s/he may have added compression with the best interests and keeping mind scenarios that s/he had encountered, the decision was definitely wrong.

Linux - Always a bridesmaid, never a bride...

2009-05-05T06:09:00-07:00

It had been a little over 4 years since I last tried out Linux. That experience wasn't half bad, but I didn't get much in the way of use out of having Linux on one of my boxes - I wasn't doing any Linux specific development and there didn't seem to be any particular programs of quality that I wanted or needed to run. One of my main uses of a computer is to play video games and Linux is a poor choice at best for video gaming compared to a Windows based machine. But with my growing interest in the audio side of game development, I figured it wouldn't be a bad time to give it another shot.

I had read that there were quite a few programs worth trying out, some of which are in use by semi-pros and amateurs in the field of digital audio / music creation and synthesis. I first tried out Ubuntu 8.04 - it wasn't too bad, but I ran into issues with the onboard network card - it just plain wouldn't work. I spent several days troubleshooting, reading various manuals, howtos, guides, forum posts, etc but no dice. I figured I ought to try out another distro before throwing in the towel and going back to WinXP. So, it was onto Fedora Core 10. Still no dice. I then slapped in a network card I scavenged off an older machine, disabled the onboard network card and was off to the races.

The installation was not too bad - odd thing was that it detected my PATA harddisk as the primary disk and the SATA one as the secondary; this is the opposite of what is specified for boot order in the BIOS and also different from what is reported by WinXP during installation. Still, got past that easily. The rest of the installation was pretty smooth and I had an FC10 Linux box up and running. That's when the problems began.

Where Money is not a Motivating Factor, Ego is.

In the world of open source / free software, most folks don't typically get paid to work on various projects - they do it primarily due to self-interest, either altruistically or not. Either way, since there isn't much in the way of monetary compensation, egos often tend to run amok. I saw a classic example of this when I ran into issues with IPV6 on FC10. Here's the long and short of the problem: One of the common libraries (glibc) that shipped with FC10 broke compatibility with well known and commonly used existing programs (firefox, wget, traceroute, etc) that are IPV6 aware. Essentially, name lookups would fail resulting in these programs not working. You could get around that in firefox by disabling IPV6 via an internal, hidden setting that most folks wouldn't know about or couldn't figure out (since they couldn't get to the internet) unless they had another computer. One would think that something as fundamental as this would have been caught in QA, but one would be dead wrong.

It wasn't that the issue was not pointed out; in fact, a beta user filed a bug one whole month before the release highlighting the problem. It was brushed off as an oddity by one of the core developers. Further more, the developer claimed that the problem was not severe enough to be patched. There were about a dozen folks clamoring to convince him with facts that the issue was severe and had to be addressed immediately. See this bug report for details (PDF[1]), if you have the patience to read through it. It's not that the developer is technically wrong - it's just that being technically right isn't the be-all and end-all of it. If your software (the core applications for the OS) doesn't work for your users, they're not going to use your OS. So, Linux loses.

Some additional blemishes are that the solution to this bug is non-trivial - certainly not something that someone without a fair amount of knowledge and expertise could figure out. I'm not a Linux expert, but I consider myself technically savvy and it took me more than a day to figure out that this was the bug I was running into. Most folks were content to disable IPV6 in firefox and get on with their lives. Before I forget, this bug renders the software update programs (yum on FC10) inoperable. So, you can't use yum to get the fix.

Here's what needs to happen to get the fix: First you must fix firefox, so you can browse the web. You then need to locate the patched library packages, which is like navigating a maze - in the dark... without any light... and some minotaurs thrown in for good measure. Once you find the patches page (PDF[2]) that lists the packages, you have to figure out from some other source(s) which of those patched packages actually apply to your installation. No, it's not obvious. Once you figure that out, you download the packages and install them using rpm - not yum, but rpm. Then everything starts working and you can get system updates, etc.

Would Your Mom Go Through This?

Or the more obvious question - would you put your mom through this? I wouldn't - my mom would kick my ass if I didn't get her computer working in 10 minutes, maybe 15 if she felt generous. ;-)

Yes, I know Windows has problems, but with money being a motivating factor, Microsoft gets most things right for most people. WinXP is a very solid operating system - was when it came out as well and has only gotten better. Yes, Vista is a slightly different story, but if you were on the inside, you'd know that most issues are due to bad drivers supplied by hardware vendors. Besides, SP1 makes it a fabulous system to work with for the most part.

Linux is just not ready to go mainstream for desktop use - not only because of debilitating bugs, but because of the attitude of the some folks in the development community towards its end users. Sad.

__Hot off the presses__: The just-released Ubuntu 9.04 distro has the same dns resolver bug (PDF[3]) too. Yay!

Vijay Talks Tech

macOs grep -f Is a Nested Loop Join; Use ripgrep or mawk Instead

The Cost Is 6.5 Milliseconds per Pattern per Mebibyte

Every Line Against Every Pattern Is a Nested Loop Join

Reach for ripgrep, Unless the Match Sits at a Known Position

The Same Command Name Is a Different Program

Fifteen Hours In, grep Had Written 2.7% of the Output

Related

Review: Brief: Make a Bigger Impact by Saying Less

Verdict: Your Mileage May Vary

The Good

The Bad

The Ugly

Notes

7 Capital Sins

Trimming: Level 1 to 3 Details

Outlines

Mind Mapping

BRIEF Maps

Controlled Conversations

Raise the Bar: Ennobling Habits

Review: Kanban in Action

Verdict: Highly Recommended

The Good

The Bad

The Ugly

Notes

Kanban Principles

Visualizing Your Work

Work Items

Work in Process (WIP)

Limiting WIP

Managing Flow

The Seven Wastes of Software Development

Five Focusing Steps

Classes of Service

Planning and Estimating

Planning Poker

Process Improvement

Retrospective - Five Distinct Parts

Metrics

Principles of Theory of Variation

Pitfalls

Review: The Life-changing Magic of Tidying Up

Verdict: Recommended. But be ready to filter out some strange anecdotes and slightly whacky techniques.

Review: Getting Results the Agile Way

Verdict: Recommended (but only chapters 1-4)

Notes (only Part I)

Foundation

10 Values

10 Principles

12 Practices

Hot Spots

Activities

Active Projects

Backlog

Monday Vision, Daily Outcomes, Friday Reflection

Monday Vision

Daily Outcomes

Friday Reflection

Review: User Story Mapping

Verdict: Recommended

The Good

The Bad

The Ugly

Review: Software in 30 Days

Verdict: YMMV (Your Mileage May Vary)

The Good

The Bad

The Ugly

Review: The Five Dysfunctions of a Team

Verdict: Recommended

The Good

The Bad

The Ugly

Notes

Pyramid of 5 Dysfunctions

1. Absence of Trust

Causes

Members of Teams with an Absence of Trust