NLOOVM

Just a sort of random thought; I think it’s interesting to see the number of efforts to build new languages on old vm’s (NLOOVM).  For example:

JVM:

  • Clojure
  • Scala
  • Kotlin

BEAM:

  • Elixir
  • Lisp Flavor Erlang (LFE)
  • Alpaca

These new languages are getting lots of traction.  Likewise it’s an interesting (and somewhat puzzling) situation with the CLR.  Pretty much anyone working on the CLR is working on C#.  Oh there are other languages on the CLR but they have nowhere near the level of interest or penetration of either Clojure or Elixir respectively.

Some Updates On “LDAP Authentication With Phoenix”

There’s an excellent write-up on authenticating users against an LDAP database with Phoenix which was created by Richard Nyström.  For some reason even though it’s barely a year old, there seem to be a few points that are either out of date or simply incorrect. I worked through his example and I wanted to share a few amendments to his original code in the interests of other developers who may want to use LDAP with Phoenix.

First the set up of Phoenix


mix phx.new ldap_example # was mix phoenix.new ldap_example
...
Fetch and install dependencies? [Yn] Y

cd ldap_example

mix ecto.create (configure your db in config/dev.exs if needed)

mix phx.gen.schema User users username:string name:string email:string
# was
# mix phoenix.gen.model User users username:string name:string email:string

mix ecto.migrate

Next change is in the session_controller.ex file:


#was LdapExample.SessionController

defmodule LdapExampleWeb.SessionController do

#was LdapExample.Web, :controller

use LdapExampleWeb, :controller
alias LdapExample.{User, Repo, Ldap}

def new(conn, _params) do
render conn, "new.html", changeset: User.login_changeset
end

def create(conn, %{"user" => params}) do
username = params["username"]
password = params["password"]
case Ldap.authenticate(username, password) do
:ok -> handle_sign_in(conn, username)
_ -> handle_error(conn)
end
end

defp handle_sign_in(conn, username) do
{:ok, user} = insert_or_update_user(username)
conn
|> put_flash(:info, "Logged in.")
|> Guardian.Plug.sign_in(user)
|> redirect(to: page_path(conn, :index))
end

defp insert_or_update_user(username) do
{:ok, ldap_entry} = Ldap.get_by_uid(username)
user_attributes = Ldap.to_map(ldap_entry)
user = Repo.get_by(User, username: username)
changeset =
case user do
nil -> User.changeset(%User{}, user_attributes)
_ -> User.changeset(user, user_attributes)
end
Repo.insert_or_update changeset
end

defp handle_error(conn) do
conn
|> put_flash(:error, "Wrong username or password")
|> redirect(to: page_path(conn, :new))
end

def delete(conn, _params) do
Guardian.Plug.sign_out(conn)
|> put_flash(:info, "Logged out successfully.")
|> redirect(to: "/")
end
end

The changes to user.ex are not made in the web/model/user.ex; rather they’re made in the existing user.ex file.

And finally a change to session_view.ex

#was LdapExample.SessionView

defmodule LdapExampleWeb.SessionView do

use LdapExampleWeb, :view

end

Quick Elixir Debugging Tip

HT: Dmitri Skliarov for sharing this tip with me.

You can quickly see the call stack being invoked when you call a given Elixir function via a quick and simple call out to the Erlang DBG module. The following tip is a very simple example of how to use this; there’s quite a bit more that can be done with DBG but you need to look at the module docs to see them.

First a little code that we want to trace:


defmodule Trace do

def add(n, m), do: n + m

end

Next you will want to initialize the debugger:


:dbg.tracer()

# => {:ok, #PID<0.66.0>}

NB: you are extremely unlikely to get the same PID I’ve shown here. Next we need to tell the debugger which item or items we want to trace:


:dbg.p(:all,:c)

# => {:ok, [{:matched, :nonode@nohost, 45}]}

Next we tell it precisely what we want to track:


:dbg.tpl(Trace,:add,:x)

# => {:ok, [{:matched, :nonode@nohost, 1}, {:saved, :x}]}

NB: the first parameter to tpl is actually an atom but since a Module name is automatically an atom it’s fine to pass the module name without a colon in front of it. Also note that you can pass the special atom :_ as the second parameter to trace all calls to all functions.

And finally we actually trace our code:


Trace.add(1,1)

# => (<0.57.0>) call 'Elixir.Trace':add(1,1)
# => (<0.57.0>) returned from 'Elixir.Trace':add/2 -> 2
# => 2

And for reference when we use :dbg.tpl(Trace,:_,:x):


Trace.add(1,1)
(<0.57.0>) call 'Elixir.Trace':'__info__'(macros)
(<0.57.0>) returned from 'Elixir.Trace':'__info__'/1 -> []
(<0.57.0>) call 'Elixir.Trace':add(1,1)
(<0.57.0>) returned from 'Elixir.Trace':add/2 -> 2
2

 

And finally when you’re done tracing:


:dbg.stop_clear()

There’s a lot of power using this technique so I’d advise that you take some time to look at the DBG man page before you start using this extensively.  Also, it’s my understanding that you should always be careful to use this technique only in development because if you trace the wrong thing you can cause deadlocks.

 

 

 

 

Quick Elixir Tip

I use Samuel Tonini’s Alchemist (and Elixir mode) to make myself much more productive with Elixir and Emacs.  But one thing that I’ve found to be a minor annoyance is that whenever I start up Emacs I have to set the scratch buffer to be in Elixir mode and I then need to change the comment characters.

Thanks to this excellent tip I no longer have to do this.  I’ve modified my ~/.emacs to add these lines to the bottom:


(setq initial-major-mode 'elixir-mode)

(setq initial-scratch-message "\
# This buffer is for notes you don't want to save, and for Elixir code.
# If you want to create a file, visit that file with C-x C-f,
# then enter the text in that file's own buffer.")

And when I get a scratch buffer I’m all good to go with Elixir!

Making Exrm Work On Windows

So as many of you will know I’m very interested in Elixir.  I also want to see it work better on Windows.  There are a few of us that have been working on it.

I’ve also been very interested in how one can deploy an Elixir app on a machine which doesn’t have Erlang and Elixir already installed.  While most of us don’t mind having to have Erlang and Elixir around because we are doing development work, it would be much more secure to only have to deploy the binaries that need to be deployed for an app.

Since I started on Elixir one of the nicest folks in the community (and that’s saying something) has been Paul Schoenfelder. When I was first struggling to learn Elixir, Paul very patiently helped me to get started.  So I became aware of one of the utilities that Paul built: EXRM.  Exrm is a tool to automate the production of a package of the needed Elixir and Erlang binaries to deploy an app. It allows a developer to deploy his or her Elixir app with only the needed runtime files so that the target machine doesn’t have to have Elixir or Erlang installed.

So I asked Paul what I could do to sort of return the favor for the help he gave me when I started and he mentioned that he wasn’t as pleased as he might be with the support for Windows in EXRM.  So I started digging into EXRM to see what I might do to get it working on Windows.

A few instructive things I spotted right away. I noticed that the way the Windows batch files were constructed they will not work correctly if they’re not run from the correct directory. I mention this for others who may try to use EXRM on Windows.

Spaces In Directory Names

I noted also that there were several places where directories containing spaces and too many characters were also a problem. The original developers of the batch files wrapped double quotes around all the directory names. If you decide to build Windows batch files, please note that wrapping double quotes around directory names is a bit error prone and tough to get right.  Hence I opted to modify all the various places where they had double quoted directory names to using 8.3 versions of the names.  That fixed up all the references to directories so that everything was found correctly.

Getting Services Working

Next there was the question of getting Windows Services working for Erlang. While digging into the batch files I discovered that basically the issue hinged on the installation of the service.  Once the service was installed correctly, it looked as if the rest of the operations on the services (starting, stopping, restarting etc.) would work correctly.

I had to hunt around for a while to figure out what I needed to do to get the service to install correctly.  I finally figured out that what was happening was the wrong parameter was getting passed as the binary to be started when the service gets started.  And so I repaired it.

A Few More Details

You’ll also need to unblock the erl.exe and epmd.exe files for the Windows Firewall.  You can wait for Windows to prompt you or you can proactively unblock them via this escript.

You may also need to deploy msvcr100.dll to your deployment machine as well.  If it’s a Windows 10 machine you may need to do this.

So we’re almost ready to roll the changes into EXRM so we can get people to be able to deploy Elixir apps on Windows.

I will continue to work with Paul on improving support for EXRM on Windows.  Keep watching this space; we’ve got some interesting ideas we’ll be trying to roll out in the next few months or so.

 

 

 

Railway Oriented Programming In Elixir

A while back Scott Wlaschin posted a great blog post on what he called “Railway Oriented Programming.”  It’s a great treatment of a common problem in adopting functional programming; how to handle a chain of functions when one of the functions in the chain may return an error.  Of course stopping the chain when the first error is returned isn’t a difficult issue; the difficulty arises in structuring the code such that the code can continue running all the functions until and unless an error is encountered.  It’s a great read and a powerful idea.

So today someone asked a question on StackOverflow about how to accomplish basically that same idea in Elixir.  A little tricky to translate Scott’s ideas to Elixir because in his article his code is specified in F# and he created a special type to capture the success or failure of a function. F# is statically typed; elixir is dynamically typed which is, of course, a substantial difference.

At any rate, I saw the question and didn’t see that Saša Jurić had already posted a great answer.  So I posted my answer and then noticed his.  I mention this because I want to be clear; I wasn’t trying to steal anyone’s thunder or trying to enhance my rep or anything like that.  I just got excited; “hey, I know the answer to that question!” so I just rushed off and hacked together some demonstration code without bothering to look any further.  I am pleased to see that the answer I came up with was pretty much the same as Saša’s; I consider him an extremely smart developer and coming up with an answer pretty similar to his  independently makes me feel a bit more confident of my own skills.

The TL;DR version is that you need to construct the functions to be applied in such a way that you can carry forward an indication of either success or failure along with the value to be checked.

defmodule Password do

  defp password_long_enough?({:ok = _atom, p}) do
    if(String.length(p) > 6) do
      {:ok, p}
    else
      {:error,p}
    end
  end

  defp starts_with_letter?({:ok = _atom, p}) do
   if(String.printable?(String.first(p))) do
     {:ok, p}
   else
     {:error,p}
   end      
  end


  def password_valid?(p) do
    {:ok, _} = password_long_enough?({:ok,p}) 
    |> starts_with_letter?
  end

end

And one would use it like so:

iex(7)> Password.password_valid?("ties")
** (FunctionClauseError) no function clause matching in Password.starts_with_letter?/1
    so_test.exs:11: Password.starts_with_letter?({:error, "ties"})
    so_test.exs:21: Password.password_valid?/1
iex(7)> Password.password_valid?("tiesandsixletters")
{:ok, "tiesandsixletters"}
iex(8)> Password.password_valid?("\x{0000}tiesandsixletters")
** (MatchError) no match of right hand side value: {:error, <<0, 116, 105, 101, 115, 97, 110, 100, 115, 105, 120, 108, 101, 116, 116, 101, 114, 115>>}
 so_test.exs:21: Password.password_valid?/1
iex(8)>

A couple of comments on the code.  Someone might ask why password_long_enough?  gets a parameter of a tuple of atom and string. Since it’s the first function in the chain it’s not really needed; all it needs is the string.  I constructed in this way for a few reasons:

  1. It’s more parallel to the other function
  2. The functions are really intended to be rules which should be checked so there should not be an order dependency. That is, they should be interchangeable.

Basically by adding any predicate function that we wish given that it takes a tuple of an atom and a string and the atom is matched against :ok, we can add any arbitrary set of tests we wish to add.

As with most things on this blog, I write this down as much for my own reference as for the edification of others who might read it.  I’m pretty sure at some point in the future, I will need to validate a value through a set of predicate functions and I’ll look at this blog post to see how I can do it.


EDIT: I was reminded of a great blog post written up by Zohaib Rauf on precisely this same idea.  Well worth reading.

Athena Code

One of the perverse side effects of code that’s “the simplest thing that could possibly work” is that it can appear much easier to create than it actually was.  I refer to this as Athena Code because it can appear as if the code sprung full-grown and ready to go from the head of the developer that created it.  In my experience as a developer this almost never happens and it can give developers the false impression that they should be able to spit out simple solutions on their first try.

A Specific Example

Recently I had a need to generate a list of all dates between a given start date and an end date in Elixir. This is the code that I actually ended up with:

defp generate_all_valid_dates_in_range(start_date, end_date) when start_date <= end_date do (:calendar.date_to_gregorian_days(start_date) .. :calendar.date_to_gregorian_days(end_date)) |> Enum.map (&:calendar.gregorian_days_to_date/1)
end

#And here's how it would work.
#iex(4)> generate_all_valid_dates_in_range({2012,1,1},{2012,2,1})
#[{2012, 1, 1}, {2012, 1, 2}, {2012, 1, 3}, {2012, 1, 4}, {2012, 1, 5},
# {2012, 1, 6}, {2012, 1, 7}, {2012, 1, 8}, {2012, 1, 9}, {2012, 1, 10},
# {2012, 1, 11}, {2012, 1, 12}, {2012, 1, 13}, {2012, 1, 14}, {2012, 1, 15},
# {2012, 1, 16}, {2012, 1, 17}, {2012, 1, 18}, {2012, 1, 19}, {2012, 1, 20},
# {2012, 1, 21}, {2012, 1, 22}, {2012, 1, 23}, {2012, 1, 24}, {2012, 1, 25},
# {2012, 1, 26}, {2012, 1, 27}, {2012, 1, 28}, {2012, 1, 29}, {2012, 1, 30},
# {2012, 1, 31}, {2012, 2, 1}]

I have deliberately omitted comments so the simplicity of the code is, hopefully, a bit more apparent. For those that don’t read Elixir, basically I take the start and end date and I convert them to Gregorian dates. I then create a range from the two Gregorian dates and pass the range as the first argument to the Enum.map function which follows.  As those of you familiar with functional programming will guess, the map takes each element in the list and performs the specified function (in this case the Erlang stdlib calendar module gregorian days to date function) on it, creating a new list with the result.  {yyyy,mm,dd} is how one specifies a date literal in both Erlang and Elixir.

By the way, many thanks to the folks on the Elixir Lang Talk mailing list for helping me to simplify my first pass at this code.

Now some may argue that that code above while simple is not particularly robust.  What happens if someone passes a bad date?  Well, that’s the Erlang way–fail fast.  If someone passes a bad date, the code will crash as soon as it’s executed. And it wouldn’t be hard to add a couple of lines of code to validate good inputs and it wouldn’t detract from the basic simplicity of the code.

Try Number 4

The point I’m trying to make is while that code seems obvious (I hope) it’s far from the first code I came up with to solve my problem.  It was actually try number 4.  I won’t share the code from tries 1 through 3 mainly because like any developer I want people to think I’m brilliant so I don’t want to keep failed experiments around.  But in general terms this is what I tried:

First try:

Add one day to the first date

Is the resulting date equal to the end date?

Yes -> Stop

No -> Add the result to the output list and loop to the top.

There was nothing particularly wrong with this approach–the code was just a lot more complicated than that sounds.

Second try:

Use a comprehension to try to generate the list

Got nowhere with this approach at all.  Totally failed idea.

Third try:

Modify the comprehension from try 2.

Again, total failure.

Fourth try:

Actually looked like this:

def generate_all_valid_dates_in_range(start_date, end_date) when start_date <= end_date do 
  (:calendar.date_to_gregorian_days(start_date) .. :calendar.date_to_gregorian_days(end_date)) 
  |> Enum.to_list
  |> Enum.map (&(:calendar.gregorian_days_to_date(&1)))
end

As I say, my fellow developers on the Elixir Talk mailing list helped me to make my code even simpler yet by pointing out that the Enum.to_list was redundant and that I could directly invoke the :calendar.gregorian_days_to_date function without having to wrap it in a lambda.

My point is this: it’s almost never easy to create “the simplest thing that could possibly work”.  Simplicity is extremely hard to create.  Almost never will your first pass be the best answer.  Don’t beat yourself up if you can’t manage “Athena Code”; almost no one can.  And don’t look at someone else’s code which is a monument to simplicity and think that that’s the way it looked when they first wrote it.  Our job as software engineers can be hard enough without putting unrealistic expectations on ourselves.

Stupid Doc Tricks In Elixir

So last night we had our first Detroit Erlang/Elixir meetup and as part of it I wanted to do a little coding exercise to help people to get their feet wet with Erlang and/or Elixir, as they chose.  We simply worked through FizzBuzz and we came up with two different Elixir solutions and two different Erlang solutions. I never stop being surprised by developer creativity.

That said, I keep striving to improve the code I write and striving to use the tools that the language provides me.  Elixir has module attributes and it’s also got module documentation as a first class citizen in the language.  So I thought to myself why not extend my simple fizzbuzz code a bit to use these two features–to better learn how to use them.  Here’s the code such as it is:


defmodule FizzBuzz do
@fizz 3
@fizzmsg "Fizz"
@buzz 5
@buzzmsg "Buzz"

@moduledoc "
Ye olde FizzBuzz test in all of its glory.
If the number is a multiple of #{@fizz} then print #{@fizzmsg}.
If the number is a multiple of #{@buzz} then print #{@buzzmsg}.
If it's a multiple of both then print #{@fizzmsg<>@buzzmsg}.
Else print the number itself.

Example usage:
iex> for i <- 1..100, do: FizzBuzz.getFB(i)
1
2
#{@fizzmsg}
4
#{@buzzmsg}
"

  def getFB(n) do
    cond do
      rem(n,@fizz) == 0 and rem(n,@buzz) == 0 ->
        IO.puts @fizzmsg <> @buzzmsg
      rem(n,@fizz) == 0 ->
        IO.puts @fizzmsg
      rem(n,@buzz) == 0 ->
        IO.puts @buzzmsg
      true ->
        IO.puts "#{n}"
    end
  end
end

Now the code itself isn’t anything very special.  But the module attributes and the fact that I can avoid specifying magic numbers is just awesome.  What if I wanted to change the code to print “Argh!” on 3 instead of “Fizz”?  Of course were I to change the number for “Fizz” then I would need to make more extensive changes but this does cover one common type of change. With the code set up in this way, I only need to change things in one place and my docs will be updated as well as my code.

I know this is sort of a “Stupid Doc Trick” but it still seemed worth sharing with others.  The idea of only needing to change a constant in one place and having everything else update is a very appealing one.

A Naive Stack Implementation In Elixir

So one of the things I’ve done recently to help me to learn a new language is to tackle creating certain data structures and common operations on those structures in a given language.  And lately I’ve been digging Elixir quite a bit.  So I decided to tackle one of the easiest things I could tackle–a simple, naive stack implementation in Elixir.  First here’s the actual Elixir code:


defmodule Stack do

defstruct elements: []

def new, do: %Stack{}

def push(stack, element) do
 %Stack{stack | elements: [element | stack.elements]}
 end

def pop(%Stack{elements: []}), do: raise("Stack is empty!")
 def pop(%Stack{elements: [top | rest]}) do
 {top, %Stack{elements: rest}}
 end

def depth(%Stack{elements: elements}), do: length(elements)
 end

# Stack.new |> Stack.push(1) |> Stack.push(2) |> Stack.pop

First off, credit where credit’s due: this is more Saša Jurić’s code than it is mine.  I posted my initial code to Code Review and a few folks gave me some great suggestions on how to improve it. But his code seemed the best implementation so this is pretty much his code.

Looking at the code a bit closer, one thing long time OO folks will notice, and something that gave me a lot of trouble initially, is the fact that there’s no member variable to stash the stack in.  The stack has to be passed into each call and the new stack is returned.  This is a big leap in logic and even now after having played with functional for a few years it can still form a large stumbling block for me when I’m trying to solve a problem.

Another thing to notice is this line:


defstruct elements: []

This is within the scope of the module but what is it doing if there’s no member data associated with the module?  It’s easy to assume that modules are analogous to classes in OO and therefore it’s also easy to assume that elements is member data–but that would be a bad assumption.  Elements is actually an abstract data type closely associated with the Stack module.  It’s a way of specifying more information about the data that the various functions in Stack will work with.  You cannot access elements from outside of Stack and in fact it’s not private data within Stack.

One comment Saša made resonates for me. To paraphrase his point,  there is already a stack implementation in Elixir; it’s called a list.  So why bother to build another one?  It’s a fair question; my answer is that if I needed a stack, I’d rather see code that deals with “Stack” directly than see code that deals with lists and has comments littered all over the place that it’s using a list as a stack.  Of course, it’s syntactic sugar but at some level anything other than raw 1’s and 0’s is syntactic sugar.  It’s a question of making it easier for other developers to understand the intent of the code.

I share this not because it’s great or interesting code (although it was much improved by feedback from Johnny Winn, Saša and José Valim) but because someone may want to study this relatively simple code to learn more about Elixir.