From C to Java to Ruby

I was having this casual conversation with a colleague about Ruby who isn’t using Ruby and I was selling him the elegance of Ruby and all that.
I said – “Remember the feeling when you moved from C development to Java? It felt like you have shed several pounds of weight attached to your shoes and you can now walk light. It is the same Deja Vu when you start to use Ruby coming from Java world.”

Then later when I was at my desk I thought of a better analogy.

• Working with C was like using an Axe, cutting away at a tree trunk. Powerful but tiring and you run the risk of missing the tree and getting your toe.

• Working with Java was like throwing away the Axe but getting a nice hack saw. It was controlled and safe. A little slow but easy to work with.

• But working with saw can be boring monotonous repetitive work. So come in Ruby. Ruby is like a Swiss army knife. Elegant, beautiful, small, doesn’t require you to wrestle with it, can be different things at different times but as the runtimes of today, much slower than Java. You may not be able to cut a tree but can carve beautiful motifs out of it.

SIP: Why is ACK for 2xx response a separate transaction?

It is widely understood that in a SIP INVITE transaction the ACK for a response other than 2xx is part of the same transaction as INVITE and the ACK for a 2xx response is a separate transaction. Every one knows this as a fact because it is written almost exactly as above in the 3261.
Not many people, however, understand why this is the case, which could be the key to better design.
ACK for a 2xx response is not generated by the transaction layer but by the UAC core and it may contains an answer to the offer received in the 2xx response.
Also since it is the goal of SIP designers to keep the transaction layer independent of the Transaction User (TU) layer, so the behavior of transaction layer on receipt of 2xx response is same in both UA and Proxy case – terminate the transaction and send the response to TU where if it is a UA it will generate an ACK and if it is a Proxy it will forward the response.
Any retransmission of the 2xx response is also dealt by the TU layer and not Transaction layer as the transaction is terminated on receipt of first 2xx response.
It is also possible that the INVITE may result in more than one 2xx response and since the first 2xx destroys the client transaction, all responses are directly dealt with by the the TU.
Because of the fact that the original transaction is destroyed to allow TU to deal with the 2xx response, the ACK is sent as part of a separate transaction.
ACK for 2xx is a subsequent request within a dialog like any other in-dialog request, except for the way the numeric value of CSeq is used, which is still the same as the original INVITE. This piece alone associates the ACK with the original INVITE at both the UAs, as there may be more than one INVITE transactions (re-INVITEs) going within the same dialog.

All this is fine and I hope it makes sense. The next logical question is if ACK is really a separate transaction then why does it not have a response?
Well, for once it is not required, because if ACK is not received by the UAS it will retransmit the 2xx response, note that this is true for any transport even reliable, as it ensures that the offer answer exchange is completed end to end.

You may have worked with RFC 3262 which introduces the reliable provisional responses. The concept is an extension of INVITE-2xx-ACK exchange in that the INVITE-ReliableProvisional-PRACK exchange makes the response reliable. However, PRACK in this case unlike ACK *does* have a response of its own. You may ask if a response for this type of transaction is not required then why is it required for PRACK, well theoretically the same arguments as for the ACK also holds for PRACK and a response is not required for reliability. However, since RFC 3262 is an extension of base SIP protocol and since base SIP protocol accords the special status only to ACK requests, it would have been a big compatibility issue for the existing infrastructure.

Coming back to ACK to a 2xx response – This is really a special transaction unlike any other; in fact in creation and sending of this the transaction machinery is hardly involved.
If you carefully read RFC 3261 then you would get a sense that there is no client transaction at the UAC and no server transaction at the UAS associated with this ACK request but the reason I say “hardly” involved is because this ACK has to have a Via header with a proper branch parameter. Remember branch parameter is required to identify a transaction. The next question to ask is - if there are no real transactions involved then why bother with a branch parameter?
The reason is that even though there is no transaction involved and the ACK to 2xx passes through the transaction layer at the UAS like any other request, only when no matching transaction is found is it handed over to TU. While at the UAC it is just a message with a new branch identifier which make it sound like a transaction but does not need to pass through the transaction layer. In fact it is directly passed to transport by TU and retransmitted when the TU gets a 2xx retransmission.
An absence of branch (if it were allowed by SIP) would have resulted in the same behavior but like almost everywhere in SIP, generality rules.

Let me close this with a tangent, to which I will come back some other time with more details.
If SIP protocol definition were an object oriented system then while designing PRACK request they could have made PRACK a subclass of ACK and so the proxies in between would not have had to learn new behavior and PRACK could have done without a response.
Unfortunately SIP (and most other protocols) is not described as object oriented system in some DSL, they are instead written in English prose.
IFIP (International Federation for Information Processing) is doing some work in the area of object oriented protocol specifications but that work is unfortunately not easily accessible.
I would have loved to see a SIP RFC besides being described in English as it is now, also accompany an interaction / behavior definition in a machine consumable form which could have been used to automatically generate parsers, state machines and higher order objects to manifest the behavior. This is very attractive but extremely difficult to realize in practice, but as I said I will revisit this thought again.

What is XML good for anyway?

XML has had a fairly successful run for almost 10 years now, shot into prominence by its “configuration” usage by Java platforms.
Sadly for XML that 10 year ride is showing signs of slowing down.

I have always disliked XML in most of its incarnations and I am extremely happy that people are now (finally) showing it the door from places where it should never even have found any foothold in the first place.

Let me recant some of what I said, XML does have some uses but for some very strange reasons it had greatness thrust upon it. Now that it is being relieved of its undeserved celebrity status, my prejudice is giving way to prudence.
So I thought of writing my honest thoughts about XML.

XML is great for hierarchical, repetitive-pattern data. It is both state and presentation clubbed into one document and that is what makes it unique. XML data is inherently de-normalized as against relational systems.

<employee>
  <name>Nasir Khan</name>
  <profession>programmer</profession>
  <address>
     <street> 235 Montgomery Street </street>
     <city> San Francisco</city>
     <country>USA</country>
  </address>
  <contact>
    <email>myemail@mydomain.com</email>
    <phone>
       <home>408-111-1111</home>
       <office>415-400-7000</office>
    </phone>
  </contact>
</employee>

<employee>
……

The above XML looks fairly standard but in a relational world this data would be constituted from several de-normalized relations (tables).

Well XMLs were never anyway designed for data storage so it’s a bad comparison. But unfortunately it was used like that in many situations.

I have tried to compile a list where XML is definitely NOT suited but unfortunately used pervasively, this is not a complete list but is based on my first hand experience.

• Information Interchange: And yes this includes Web services and SOAP. They could have designed much simpler protocols for data exchange, markup just adds unnecessary clutter and wastes bandwidth. XML for data/information exchange is neither human readable nor machine friendly. “Extensible protocols” is an oxymoron, only when both client and server “understand” the protocol is when it is usable, so what is extensible about it? Go figure. The above data snippet (the record) could well be coded in a simple delimited string “Nasir Khan, programmer, 235 Montgomery street, San Francisco, USA, myemail@mydomain.com, 408-111-1111, 415-400-7000”.
  This is 115 characters as against 313 characters in XML form! If a machine is reading the string then believe me it does not care about angled brackets, it does what you tell it to do. If you tell it that the second record in the string is my profession it will just take my word for it.
  What about readability? Take a step back and ask someone who is not a programmer about which of these make sense to them. Even a school dropout will be able to make sense out of the delimited string, while most will cringe at the sight of that ungainly XML. The context (schema) is coded in human brains.
  I know you could argue – “how could you tell the difference between home and office phone”, well just prefix a “h:” or “w:” before the number and off you go.
  Bottomline - they could always have designed simpler text protocols which could map to object oriented systems (like CORBA IDLs or other schemes) for automatic generation of protocol handlers on automatons and save millions in bandwidth, time and eye sores (the pointy arrows really hurt).



• Programming languages/DSLs: When I first saw constructs like
  
  

  
  <if>value</if>
    <then>
   …..
   <and>
    <or>
      ….
    </or>
    <or>
     …..
    </or>
  </and>
  

  
  I just asked “WHY?”
  I still do not understand why there are so many DSLs out there based on XML.
  
  

  
  <assign>
    <copy>
      <from variable="c1"/>
      <to variable="c2"/>
    </copy>
  </assign>
  

  
  This snippet above is an example of assignment in BPEL.This is equivalent to
  

  
  c2 = c1 
  

  
  in any of the “normal” programming languages.
  Now if one were paid by the lines of code they write then it is a potential goldmine for developers but other than that it is a recipe for carpal tunnel syndrome.
  
  Even though “L” in XML stands for language, it is not a language in any sense of the word. It is a document format. People have tried to patch on language constructs like conditions, iteration constructs, recursion, assignment etc on it but it doesn’t work most of the time.
  Well theoretically you could write a full blown programming language in XML syntax. It will be utterly unreadable and make you see your chiropractor but what I think will be its nemesis is the fact that XML originally was designed to format data. There is no inherent concept of data types. Everything is a string, you could again argue that you will create tags for data types and use them to wrap your variables like <int>variable</int>, well sure but then how do you deal with collections, how would you create a tree data structure in that language?
  You will say that I can create a <tree> declaration and represent the actual elements in internal data structures. Good for you but you will then represent your program state in non-XML format while your program logic in XML!
  Remember XML is a data format structure; it will be like tacking wheels to your boat while trying to make your car float.
  
  I think the reason why people started writing DSLs in XML was that they wanted some dynamic evaluation to be done in static languages of the age like C++, Java etc. XML was popular at that time and seemed like a natural choice.
  
  In those days it went like –
  
  “Hey I am not on talking terms with my grandmother; she doesn’t talk to me at all!”
  
  “Have you tried XML? It might work. It’s hip”.
  
  That they could have done dynamic evaluation in hundreds of other ways got trampled under the XML gold rush.
  An embedded Java compiler, like the JSP processor is the best solution in most cases where you want such dynamic behavior. I am surprised why that pattern was just locked up in (immensely successful) web containers.
  



• Build tools/Ant: I never liked Ant, that is not to say I did not use it. I did, as it was the only option to build Java projects for a long time. Anyone who has used Ant would agree that after your project has reached a critical mass the build files look clunky and un-navigable. In his article Martin Fowler describes the Rake build tool and describes the flaws that an Ant like XML based system would have, primarily because of the choice of language, namely XML. I do not want to repeat what he said in his article, so have a look at Martin Fowler's article on Rake
  



• Configuration files: This is perhaps the most pervasive use of XML. I do not have too much problem with it except that the large configuration files are unreadable. The other problem is that since XML enforces a certain order, even for unrelated nodes, you have to be aware of the DTD or schema before you make any changes to the configuration. For most situations a simple Properties file or Java Preferences like access would have solved the problem nicely, particularly because the property names can very well be mapped to Java package scoped constants or variable for readability.
  
  e.g.
  

  
  com.mycompany.employee.name=”Nasir”
  com.mycompany.employee.phone=”415-111-1111”
  com.mycompany.threadmanager.numThreads=5
  com.mycompany.concurrency.policy=”Pessimistic”
  

  
  
  Is far more readable than
  

  
  <?xml version="1.0"?>
  <config-data>
   <employee>
     <name>Nasir</name>
     <phone>phone<phone>
   </employee>
   <thread-manager>
   …
   <num-threads>5</num-threads>
  </thread-manager>
   
  <concurrency-policy>Pessimistic</concurrency-policy>
  </config-data>
  

  
  What is more is that in the XML case you need repetitive Java code to parse the XML through some expensive parser and maintain Java objects representing the configuration. On the other hand with Properties like access you have built in Java class java.util.Properties to manage the data with as simple an API as load() and save().
  
  It was for Java’s heavy reliance on fat XML configuration files that prompted the detractors to coin the term “XML sit ups”.
  


• XSLT: The concept is pretty powerful. Have a base document and transform it according to the presentation needs. In fact this could be seen as a variation of the Builder Design pattern or Abstract Factory as the transformation could be based on the type of request or type of client etc.
  So what is wrong with it? It is the “X” in XSLT. Several years ago I wrote some XSLT scripts to convert XML to Java code, if you haven’t guessed what was I trying to do - I was converting a DSL in XML to plain (and simple) Java code for performance reasons. When I finished that project I had pulled a lot of my hair out, it was ridiculously difficult to do something simple. I added a todo note to rewrite that transformer in Java itself using a SAX parser rather than use XSLT.

So what is XML good for in real world? I would say a number of things, definitely not as many as it is used for but enough.

• Hierarchical data representation and XPath: One thing that I instantly liked in the XML world was XPath expressions. It is a powerful mechanism that actually uses the XML representation in the most intuitive form. As I said in the beginning, the most important feature of a XML document is its hierarchical representation. Xpath leverages that by using a familiar “path” syntax.
  Even to some one who doesn’t know Xpath, an expression like
  

  
  //place/coordinates/latitude/text()
  

  
  will instantly make sense.
  There is a protocol called XCAP (XML Configuration Access Protocol) Jonathan Rosenberg's page on XCAP used in the SIP/SIMPLE world that very cleverly uses Xpath in conjunction with HTTP to access and modify hierarchical data.
  



• GUI description: I think more than anything this is where XML truly excels. The most ubiquitous usage of XML as GUI description tool is of course HTML. If you have worked with building GUIs then you will immediately realize that GUI development is so much more complicated than “faceless” non UI code, there is not just behavioral aspect as any other program but also presentation aspects.
  XML based UI design frameworks take the pain out of UI development. Again this is possible because XML is hierarchical and presentation oriented. Widgets can be placed on the screen depending upon their position in hierarchy.
  

  
  <button-group name=”mygroup”>
    <radio-button name=”b1”, onSelect=”func1()>Male </radio-button>
    <radio-button name=”b2”, onSelect=”func2()>Female </radio-button>
  </button-group>
  

  
  Is so much more intuitive than
  

  
      JRadioButton maleButton = new JRadioButton(“Male”);
      JRadioButton femaleButton = new JRadioButton(“Female”);
  
      //Group the radio buttons.
      ButtonGroup group = new ButtonGroup();
      group.add(maleButton);
      group.add(femaleButton);
  
      //Register a listener for the radio buttons.
      maleButton.addActionListener(this);
      femaleButton.addActionListener(this);
  

  
  This is not to in any way belittle the excellent work done by Java UI team. Java APIs are great for portable UI code and very exhaustive too. To make these APIs more accessible to visually oriented developers there are a number of XML to Java projects underway. I think they hold a great deal of promise for Java UI development.
  http://www.java-source.net/open-source/xml-user-interface-toolkits
  
  There are a number of other XML based UI design languages, notable amongst them are XUL from Mozilla, MXML from Adobe, LzXML from OpenLazlo and UIML which has an open source Java binding http://sourceforge.net/projects/juiml.
  



• Annotations/Sematic tagging: A lesser known but widely used feature of XML is tagging. How many times while replying an email message you have written your reply “inline” of the original message? The way you do it plain text is to enclose your replies in tags. Like I use may name as tags
  

  
  <nasir>Yes I will attend the meeting. </nasir>
  

  
  Now the resulting email response may not be a well-formed XML document but the notion of start and end tag is pretty powerful.
  On another occasion while compiling the specification ( JSR 289 EDR document) from a large number of smaller sub-documents I used simple markup tags. I inserted tags in the source documents and then had destination tags in the main documents where the text was pulled from the sources. Besides this there were a few more things like automatic linking and aliasing but overall it greatly simplified the document creation for me.
  

To summarize XML can be a great help in some situations but can be an excruciating pain in others, perhaps this is true for any technology but in this article I have focused on XML. Based upon my experience I can tell you to keep the following in mind while using XML in your projects.

1. Checkout the alternatives like YAML, which is a document format http://www.yaml.org/ that is a very simple text based format and has object bindings in several languages. Instead of truckloads of XML and heavy parsers on two sides of web services world, you could easily have YAML going between a Java and C++ process with straightforward object binding on both sides.


2. Think in terms of simple text protocols, look at existing ones like HTTP if you are writing a client server system rather than inventing an XML based protocol. If your data is hierarchical have a look at XCAP.


3. See if your need for Java configuration can be satisfied by Java Properties, Preferences or simple YAML.


4. If you are building complex Java projects have a look at JRake in place of Ant, which takes the pain out of build management process. Particularly if you are starting afresh and do not have Ant investment done already.


5. If your data is hierarchical then XML representation may make sense, but don’t get overboard, instead use builder/transformer pattern to generate XML when needed but maintain data in a format that is best suited for your requirements.


6. And last but not the least exercise good judgment in designing with XML, question established beliefs. Think objectively if XML is the best tool for this problem. Answer the questions around performance, readability, maintenance as your XML document grows, and if you have tooling to help you with your XML needs.

Calculate once Cache forever

In the pickaxe book there is a section which talks about the use case of a "once" directive. So you have a method with complex calculations and you want to be able to cache the result.
One "clunky" technique mentioned there is to store the result in an instance variable and check it before doing the calculation.

def as_string
  unless @string
    # complex calculation
    @string = result
  end
  @string
end

The book then introduces Tadayoshi Funaba's once directive implementation.

def once(*ids)
  for id in ids
    module_eval <<-"end;"         
      alias_method :__#{id.to_i}__, :#{id.to_s}           
      def #{id.to_s}(*args, &block)           (@__#{id.to_i}__ ||= 
        [__#{id.to_i}__(*args, &amp;block)])[0]         
      end       
    end;     
  end   
end 

(For details please look at Funaba San's once )

There is a related technique that I wanted to talk about. This is slightly different in semantics as it rewrites the method itself and though cannot be a substitute for the once directive but could be used in this or similar situations.
Let's look at the code first.

class Test
def calc
  # complex calculation
  @result = "result"
  puts "Whew! Here is the result"
  if @result
    def self.calc
      puts "No sweat!"
      @result
    end
  end
  @result
end
end

What is happening here is that when the class is evaluated the outer method calc() is defined, but when the method calc() is executed, conditionally the singleton method calc() is defined on the object. The "self" here would refer to the object of the class Test in whose context the outer method calc() is executed. The end result is that we get a singleton method.

irb(main):077:0> t1 = Test.new
=> #
irb(main):078:0> t1.calc
Whew! Here is the result
=> "result"
irb(main):079:0> t1.calc
No sweat!
=> "result"

irb(main):084:0> t2 = Test.new
=> #
irb(main):085:0> t2.calc
Whew! Here is the result
=> "result"
irb(main):086:0> t2.calc
No sweat!
=> "result"

This can be used to conditionally re-define the behavior of a class on an object by object basis and so has a more general purpose use than just result caching.

How do you swap variables?

At least some 10 years ago someone asked me this question in an interview - "How do you swap two variables without using a temporary variable"
I now vaguely remember how I answered the question then but I know I did *not* use XOR in that answer, I did not know it at that time but its really very cool.

  x = x ^ y
  y = x ^ y
  x = x ^ y

But of course in Ruby you would just do

  a,b = b,a

and thats it! Have a nice day. May I help the next customer in line?

This is Ruby parallel assignment in action which has several cool uses but begins to really shine when you splat and unsplat.