Three Rules Of All Testing

There are three rules applicable to all testing environments.  I’ve articulated one of the rules before but now I’ll add two more.

Rule 1 – Dead Simple

Tests must be dead simple to run.  It doesn’t matter what you’re testing.  It doesn’t matter if you’ve got 100% code coverage. It doesn’t matter if you’ve tested everything from the smallest unit to the largest UI.  If the tests aren’t easy to run the people who most need to run them that is the developers and the analysts will figure out ways to game the system and avoid running the tests.  The rule is that running tests must be easier than any workaround the developers and analysts can come up with to avoid running the tests.

Rule 2 – Some Tests Are Better Than No Tests

I can’t count the number of times we’ve been discussing testing and someone will say “well, we can’t test X” with the implication being that since we can’t test everything there’s nothing to be gained from any testing.  This is analogous to saying the only way we can engineer a rocket is to build the whole thing and then fire it off.  Never mind building prototypes or doing any of that silly math to check our assumptions.

Now, granted there can be some cases where certain properties are so essential to a system that if they can’t be tested it really isn’t worth testing at all.  But those are extremely rare cases.  Even if you can only run tests on 50% of your code, that’s still 50% that’s tested that would not get tested otherwise.

Rule 3 – Don’t Test the Language or the Libraries

I’ve seen unit tests where someone will set some property of a class.  Then they will immediately read the property to check that is hasn’t changed.  While it’s a valid test to check an invariant on a class, it’s not valid to test it immediately after its set.  If you set a value and then read it and it’s different, you’ve got problems more substantial than a unit test will ever solve.

These are just my big three rules for testing.  I’d be interested to hear the thoughts of others on this subject.

Which Is Better A or B?

It often surprises me that people who spend their whole day focusing on details, software developers, so often ask question in broad and generic terms.  Here’s some that I’ve heard over the years:

  • Why should I learn Object Oriented Programming?  (I have been writing software for a while.  Long enough to remember the days before we discovered that OO was the only right way to write software.)
  • Won’t having a VM with Garbage Collection be really slow?  (This question came up a lot when Java was just being rolled out.)
  • Isn’t C# just Java with different keywords? (When C# was first rolling out.)
  • Why should I learn functional programming?  (One of the more recent vague and generic question I’ve gotten tons of.)

And today:

  • Will I be more productive if I write a website in Elixir/Phoenix than if I write it in (Web Framework X)?

Now consider some of the implied assumption in those questions.  All other things being equal will B (new technology) be better than A (which I’m already using)?

Remember we are discussing human beings performing a human activity–writing software.  Of anything humanity has ever tried to measure, quantify and generally treat in an empirical way, software development seems to be one of the least amenable to analysis.  Just some of the implied parameters of some of those questions above are:

a.) Is B more productive than A?

How do you measure “productive”?  Economists and experts on the subject of productivity don’t agree about measuring productivity in terms of hard goods (cars, appliances, etc.) so how should we measure productivity in terms of something we can’t even really measure?

b.) Will B allow me to create better software than A?

Define “better”.  Better in what sense?  Less errors?  Runs faster?  Less source code?  Easier for maintenance developers to understand?  More secure against hacking?

The point is this: while we wish we could answer the question of whether a new technology is better than an existing technology at all there’s so much ambiguity in so many facets of what we’re discussing there’s no way to even frame these questions in a way they can be discussed intelligently.  So for goodness sake–look for yourself and make your own decision.

Railway Oriented Programming In Elixir

A while back Scott Wlaschin posted a great blog post on what he called “Railway Oriented Programming.”  It’s a great treatment of a common problem in adopting functional programming; how to handle a chain of functions when one of the functions in the chain may return an error.  Of course stopping the chain when the first error is returned isn’t a difficult issue; the difficulty arises in structuring the code such that the code can continue running all the functions until and unless an error is encountered.  It’s a great read and a powerful idea.

So today someone asked a question on StackOverflow about how to accomplish basically that same idea in Elixir.  A little tricky to translate Scott’s ideas to Elixir because in his article his code is specified in F# and he created a special type to capture the success or failure of a function. F# is statically typed; elixir is dynamically typed which is, of course, a substantial difference.

At any rate, I saw the question and didn’t see that Saša Jurić had already posted a great answer.  So I posted my answer and then noticed his.  I mention this because I want to be clear; I wasn’t trying to steal anyone’s thunder or trying to enhance my rep or anything like that.  I just got excited; “hey, I know the answer to that question!” so I just rushed off and hacked together some demonstration code without bothering to look any further.  I am pleased to see that the answer I came up with was pretty much the same as Saša’s; I consider him an extremely smart developer and coming up with an answer pretty similar to his  independently makes me feel a bit more confident of my own skills.

The TL;DR version is that you need to construct the functions to be applied in such a way that you can carry forward an indication of either success or failure along with the value to be checked.

defmodule Password do

  defp password_long_enough?({:ok = _atom, p}) do
    if(String.length(p) > 6) do
      {:ok, p}
    else
      {:error,p}
    end
  end

  defp starts_with_letter?({:ok = _atom, p}) do
   if(String.printable?(String.first(p))) do
     {:ok, p}
   else
     {:error,p}
   end      
  end


  def password_valid?(p) do
    {:ok, _} = password_long_enough?({:ok,p}) 
    |> starts_with_letter?
  end

end

And one would use it like so:

iex(7)> Password.password_valid?("ties")
** (FunctionClauseError) no function clause matching in Password.starts_with_letter?/1
    so_test.exs:11: Password.starts_with_letter?({:error, "ties"})
    so_test.exs:21: Password.password_valid?/1
iex(7)> Password.password_valid?("tiesandsixletters")
{:ok, "tiesandsixletters"}
iex(8)> Password.password_valid?("\x{0000}tiesandsixletters")
** (MatchError) no match of right hand side value: {:error, <<0, 116, 105, 101, 115, 97, 110, 100, 115, 105, 120, 108, 101, 116, 116, 101, 114, 115>>}
 so_test.exs:21: Password.password_valid?/1
iex(8)>

A couple of comments on the code.  Someone might ask why password_long_enough?  gets a parameter of a tuple of atom and string. Since it’s the first function in the chain it’s not really needed; all it needs is the string.  I constructed in this way for a few reasons:

  1. It’s more parallel to the other function
  2. The functions are really intended to be rules which should be checked so there should not be an order dependency. That is, they should be interchangeable.

Basically by adding any predicate function that we wish given that it takes a tuple of an atom and a string and the atom is matched against :ok, we can add any arbitrary set of tests we wish to add.

As with most things on this blog, I write this down as much for my own reference as for the edification of others who might read it.  I’m pretty sure at some point in the future, I will need to validate a value through a set of predicate functions and I’ll look at this blog post to see how I can do it.


EDIT: I was reminded of a great blog post written up by Zohaib Rauf on precisely this same idea.  Well worth reading.

Configuring A WordPress Site On EC2

There’s an excellent and practical write-up on configuring an EC2 container to host a WordPress site here.  But there are a few minor amplifications I’d add.  So I thought I’d write them up here.

1.) As some folks noted in the comments, you should assign the Elastic IP address before you install WordPress.  Everything with WP just seems to work much better when you do it in that order.

2.) You’ll want to configure your instance to autostart both Apache and Mysql.  Credit where it’s due: I got this information from here.

  • sudo /sbin/chkconfig –add mysqld
  • sudo /sbin/chkconfig –list mysqld
  • sudo /sbin/chkconfig mysqld on
  • sudo /sbin/chkconfig –add httpd
  • sudo /sbin/chkconfig –list httpd
  • sudo /sbin/chkconfig httpd on

You really don’t need the –list step in either case.  It’s just a way to insure you’ve got chkconfig command right. Note that those commands are specific to the CentOS image that Amazon provides.  If you’re hosting a different flavor of Linux, you’ll need to figure out the commands for your flavor.

Other than those two minor corrections, his guide is sound and well-written.

Interesting And Unexpected Benefit Of Pattern Matching

So recently I’ve been working on automating some interactions with a website.  Elixir’s Hound library is a great help, of course.  But I also ran across an interesting and unexpected benefit to pattern matching that greatly simplified some code.

I was trying to dynamically build a URL to pass parameters. The URL would be in the form:

http://www.exampledomain.com?param1=p&param2=q&param3=r

Whenever I see code like this, it’s always a little tricky because you want to add the & at the end of each parameter except the last.  This always used to entail writing a special case in the loop to test if the index of the current element was the maximum index.  But with pattern matching and recursion there’s a much simpler and cleaner solution.  Like this:

defp add_params_to_url(url,[%{:name => name, :value => value}]) when is_binary(url) do     #1
 "#{url}#{name}=#{value}"
end
defp add_params_to_url(url,[%{:name => name, :value => value} | t]) when is_binary(url) do   #2
 url = "#{url}#{name}=#{value}&"
 add_params_to_url(url,t)
end
defp add_params_to_url(url,[]) when is_binary(url), do: url   #3

This is Elixir for those unfamiliar with it.  Basically it’s three function clauses.  The first clause (#1) is hit if only one element is in the list passed into the function. The second clause (#2) is hit if multiple elements are present in the list and the third clause (#3) is hit if the list is empty.

So this is how this works.  When I call add_params_to_url, Elixir will automatically try to match the correct function call dependent on which list I pass.  It will also stop at the first function clause which matches so that other function clauses will not be evaluated.

If I pass a list of URL params which has only one element, clause 1 is matched and the parameter and value are appended and I’m done.  The URL already has the ? on the end, of course.  If I pass a list of URL params with mutliple elements, clause 1 doesn’t match so Elixir jumps down to clause 2 and that matches.  Clause 2 takes the first element from the list tacks it on to the list of parameters with a & at the end and then recursively calls itself with the tail of the list.  If the tail of the list contains more than one parameter, clause  1 will once again fail to match and clause 2 will match again.  If it contains only one parameter, clause 1 is matched.  Because clause 1 doesn’t call itself recursively, clause 3 should never be matched; it’s actually more of a guard case in case someone calls the function with an empty parameter list by mistake.

This, to me, is a small bit of genius.  Much simpler to get exactly the effect I want, that is not having an extraneous character tacked to the end of the string, and it’s nice and simple.

By the way, it occurs to me that I might be able to use a reduce or a fold function to achieve this same effect.  The issue there is that it’s less apparent what it is that I’m trying to do and I also run into the same problem–that is, how do I deal with that last element in the list so I don’t get my separator appended?

New Versions Of Chocolatey NuGet Packages For Elixir and Erlang

Quite some time ago, I took over maintenance of the Chocolatey NuGet (CNG) packages for both Elixir and Erlang.  I did release new versions of both packages as new versions of Erlang and Elixir became available but I didn’t do a lot of work on either of them.  That changed a little bit today.

One thing that had always kind of annoyed me was the fact that I could take my pick; I could modify people’s paths on their machines without asking or I could tell people that after they installed Erlang and Elixir they’d need to fix up their paths to get Elixir to work “out of the box”.  Neither option seemed particularly good but I decided to go with the latter because I don’t believe in messing around with other people’s machines without explicit permission and I didn’t see anything in the CNG framework to prompt them.

Recently the question of why Elixir doesn’t work “out of the box” came up yet again.  I gave my standard answer and then one of the CNG mods mentioned an option that I’d not heard of before.  CNG now has the ability to create a small shim file to run another command. I say “now has” but this feature may have been in CNG all along; maybe I just never heard of it.

Regardless this is a great approach.  I can create these shims to point to the right bat files and the right exe’s, put the shims in the Chocolatey bin directory which is added to the path by Chocolatey and I get behavior much closer to the behavior that apt-get and homebrew users have been getting for a long time.

So today, I’m pushing up slightly modified CNG packages for both Elixir and Erlang.  Besides adding the shims I also removed some dead code that I had simply been too lazy to remove and I also modified the Erlang uninstall process to fix a long-standing issue.

One thing some people may take some issue with: I have called the shim to iex.bat ielixir. I did this because iex is a command in Powershell and if you’re on Windows, you should be using Powershell for your terminal.  Note that since this is simply a shim, iex.bat is still present and can still be called if anyone cares to.  I realize that on Mac and Linux it’s iex but I think this is important enough to allow this small divergence.  Maybe we could get Mac and Linux to add an ielixir alias to their packages.  After all iex is short for “interactive elixir” so ielixir isn’t totally unreasonable.

I would appreciate it if anyone who uses CNG to install Elixir would take a look at this new package and beat it up a bit.  If you find issues, please let me know via the Github repo. 1.0.5 and 18.0 will continue to be on CNG if you don’t care to make this change.

The packages are Elixir 1.0.5.20150709 and Erlang 18.0.20150709

Always the way; upload packages, let people know and then realize you forgot to test something.  The corrected packages are Elixir 1.0.5.20150710 and Erlang 18.0.20150709.2

Thanks to Gary Ewan Park and Rob Reynolds for their help in figuring out the problems I was having with creating the shim files I needed.

Some Thoughts On Windows 10

Recently I saw a great post on Extreme Tech about the upcoming Windows 10 release and, to put it in my own words, the heart of the problem with both Windows 8.x and Windows 10.  Well, one of the problems with the great “Desktop/Tablet Convergence” that Microsoft has been trying to foist on all of us since Windows 8.0.  I think that the author of that article, David Cardinal, got at the essential problem with the whole convergence strategy that Microsoft was (and still is) pursuing: the fact that people fundamentally don’t want the convergence that MS is offering.

Now I’ve seen the posts about Windows 10 supporting Android and iOS apps natively.  While I’d like to be charitable and assume that MS has decided to actually care about its users rather than its bottom line, I don’t believe that’s happened.  I see this instead as a rather cynical ploy to further fragment Android and maybe to start to try to fragment iOS.  Given the level of control that Apple exercises over iOS, I doubt Microsoft will get far in fragmenting it.  But on the Android side, I can believe they’d have high hopes of fragmenting the platform further.  After all they’re backing Cyanogen which is “going to put a bullet through Google’s head“.

The thing is, I’m still not sure who they think is going to build (or even modify) Android or iOS apps for Windows 10.  I mean they’ve offered money for developers to build apps for Win Phone and yet the platform still lacks many of the most popular mobile apps. And telling developers that there’s yet another platform for them to test their Android or iOS app on for what’s likely to be a miniscule market share–well I’m sure developers will be lining up to invest in that (not really).  Now some of you will read this and say “Miniscule market share?  What about all those Windows 7 and 8 boxes which people will upgrade to Windows 10?”  And, yes, that’s true.  But in terms of the Android and iOS market, Windows 10 will still be a drop in the bucket.  I will be shocked if that changes.

I guess, in spite of the fact that I’ve seen it time and again, I’m still surprised that people have such short memories. I can recall the days of DOS when people complained that the interfaces of apps weren’t the same from one app to another.  “It should be like a car!  Every app should work the same!” they wailed. But what was true then is still true today; when two things serve two fundamentally different objectives, they need different interfaces.  By the way, lest anyone think I don’t know my history: when Windows came along some of the superficial UI differences were removed by Microsoft. But they were superficial differences. Tablets serve a fundamentally different purpose than laptops and desktops.  Trying to force convergence isn’t going to change the fundamental issue.  Cars and forklifts are different if similar machines with different purposes and therefore different affordances.

I just wish someone at Microsoft would realize that trying to force a single OS on us for tablets, gaming consoles and laptop/desktop devices is simply not going to work.

The Detail-Oriented Nature Of Software Development

One thing I have noticed about myself over the years; I’m not as good as I would care to be at focusing on details. In fact, sometimes I’m awful at paying attention to the details. As a software developer, of course, details are extremely important.

I’ve come up with (and read about) a few tricks/heuristics to keep me from forgetting important details:

  1. Checklists
  2. Scripts/code to automate repetitive chores (i. e. the more I can put into a script the less I need to pay attention to).
  3. Rubber duck debugging (i. e. sometimes just the act of talking through a problem helps me to discover an overlooked detail).

I would like to know what other techniques and tactics other developers do to cope with the amount of detail that we deal with in the course of our job.  Please share any thoughts/ideas you have on this subject in the comments.

Athena Code

One of the perverse side effects of code that’s “the simplest thing that could possibly work” is that it can appear much easier to create than it actually was.  I refer to this as Athena Code because it can appear as if the code sprung full-grown and ready to go from the head of the developer that created it.  In my experience as a developer this almost never happens and it can give developers the false impression that they should be able to spit out simple solutions on their first try.

A Specific Example

Recently I had a need to generate a list of all dates between a given start date and an end date in Elixir. This is the code that I actually ended up with:

defp generate_all_valid_dates_in_range(start_date, end_date) when start_date <= end_date do (:calendar.date_to_gregorian_days(start_date) .. :calendar.date_to_gregorian_days(end_date)) |> Enum.map (&:calendar.gregorian_days_to_date/1)
end

#And here's how it would work.
#iex(4)> generate_all_valid_dates_in_range({2012,1,1},{2012,2,1})
#[{2012, 1, 1}, {2012, 1, 2}, {2012, 1, 3}, {2012, 1, 4}, {2012, 1, 5},
# {2012, 1, 6}, {2012, 1, 7}, {2012, 1, 8}, {2012, 1, 9}, {2012, 1, 10},
# {2012, 1, 11}, {2012, 1, 12}, {2012, 1, 13}, {2012, 1, 14}, {2012, 1, 15},
# {2012, 1, 16}, {2012, 1, 17}, {2012, 1, 18}, {2012, 1, 19}, {2012, 1, 20},
# {2012, 1, 21}, {2012, 1, 22}, {2012, 1, 23}, {2012, 1, 24}, {2012, 1, 25},
# {2012, 1, 26}, {2012, 1, 27}, {2012, 1, 28}, {2012, 1, 29}, {2012, 1, 30},
# {2012, 1, 31}, {2012, 2, 1}]

I have deliberately omitted comments so the simplicity of the code is, hopefully, a bit more apparent. For those that don’t read Elixir, basically I take the start and end date and I convert them to Gregorian dates. I then create a range from the two Gregorian dates and pass the range as the first argument to the Enum.map function which follows.  As those of you familiar with functional programming will guess, the map takes each element in the list and performs the specified function (in this case the Erlang stdlib calendar module gregorian days to date function) on it, creating a new list with the result.  {yyyy,mm,dd} is how one specifies a date literal in both Erlang and Elixir.

By the way, many thanks to the folks on the Elixir Lang Talk mailing list for helping me to simplify my first pass at this code.

Now some may argue that that code above while simple is not particularly robust.  What happens if someone passes a bad date?  Well, that’s the Erlang way–fail fast.  If someone passes a bad date, the code will crash as soon as it’s executed. And it wouldn’t be hard to add a couple of lines of code to validate good inputs and it wouldn’t detract from the basic simplicity of the code.

Try Number 4

The point I’m trying to make is while that code seems obvious (I hope) it’s far from the first code I came up with to solve my problem.  It was actually try number 4.  I won’t share the code from tries 1 through 3 mainly because like any developer I want people to think I’m brilliant so I don’t want to keep failed experiments around.  But in general terms this is what I tried:

First try:

Add one day to the first date

Is the resulting date equal to the end date?

Yes -> Stop

No -> Add the result to the output list and loop to the top.

There was nothing particularly wrong with this approach–the code was just a lot more complicated than that sounds.

Second try:

Use a comprehension to try to generate the list

Got nowhere with this approach at all.  Totally failed idea.

Third try:

Modify the comprehension from try 2.

Again, total failure.

Fourth try:

Actually looked like this:

def generate_all_valid_dates_in_range(start_date, end_date) when start_date <= end_date do 
  (:calendar.date_to_gregorian_days(start_date) .. :calendar.date_to_gregorian_days(end_date)) 
  |> Enum.to_list
  |> Enum.map (&(:calendar.gregorian_days_to_date(&1)))
end

As I say, my fellow developers on the Elixir Talk mailing list helped me to make my code even simpler yet by pointing out that the Enum.to_list was redundant and that I could directly invoke the :calendar.gregorian_days_to_date function without having to wrap it in a lambda.

My point is this: it’s almost never easy to create “the simplest thing that could possibly work”.  Simplicity is extremely hard to create.  Almost never will your first pass be the best answer.  Don’t beat yourself up if you can’t manage “Athena Code”; almost no one can.  And don’t look at someone else’s code which is a monument to simplicity and think that that’s the way it looked when they first wrote it.  Our job as software engineers can be hard enough without putting unrealistic expectations on ourselves.

A Naive Stack Implementation In Elixir

So one of the things I’ve done recently to help me to learn a new language is to tackle creating certain data structures and common operations on those structures in a given language.  And lately I’ve been digging Elixir quite a bit.  So I decided to tackle one of the easiest things I could tackle–a simple, naive stack implementation in Elixir.  First here’s the actual Elixir code:


defmodule Stack do

defstruct elements: []

def new, do: %Stack{}

def push(stack, element) do
 %Stack{stack | elements: [element | stack.elements]}
 end

def pop(%Stack{elements: []}), do: raise("Stack is empty!")
 def pop(%Stack{elements: [top | rest]}) do
 {top, %Stack{elements: rest}}
 end

def depth(%Stack{elements: elements}), do: length(elements)
 end

# Stack.new |> Stack.push(1) |> Stack.push(2) |> Stack.pop

First off, credit where credit’s due: this is more Saša Jurić’s code than it is mine.  I posted my initial code to Code Review and a few folks gave me some great suggestions on how to improve it. But his code seemed the best implementation so this is pretty much his code.

Looking at the code a bit closer, one thing long time OO folks will notice, and something that gave me a lot of trouble initially, is the fact that there’s no member variable to stash the stack in.  The stack has to be passed into each call and the new stack is returned.  This is a big leap in logic and even now after having played with functional for a few years it can still form a large stumbling block for me when I’m trying to solve a problem.

Another thing to notice is this line:


defstruct elements: []

This is within the scope of the module but what is it doing if there’s no member data associated with the module?  It’s easy to assume that modules are analogous to classes in OO and therefore it’s also easy to assume that elements is member data–but that would be a bad assumption.  Elements is actually an abstract data type closely associated with the Stack module.  It’s a way of specifying more information about the data that the various functions in Stack will work with.  You cannot access elements from outside of Stack and in fact it’s not private data within Stack.

One comment Saša made resonates for me. To paraphrase his point,  there is already a stack implementation in Elixir; it’s called a list.  So why bother to build another one?  It’s a fair question; my answer is that if I needed a stack, I’d rather see code that deals with “Stack” directly than see code that deals with lists and has comments littered all over the place that it’s using a list as a stack.  Of course, it’s syntactic sugar but at some level anything other than raw 1’s and 0’s is syntactic sugar.  It’s a question of making it easier for other developers to understand the intent of the code.

I share this not because it’s great or interesting code (although it was much improved by feedback from Johnny Winn, Saša and José Valim) but because someone may want to study this relatively simple code to learn more about Elixir.