A while ago, I started exploring Scala and wrote a post called Head Smacking in Scala on Positively Glorious! (Republished here for convenience). Yesterday I found another head smacking problem that taught me a few things.

Recall that I was creating an XML parser to pull metadata and results from USGS Stream Gauge sites.¹ Yesterday, I decided to explore some of Scala’s DBC abilities so that I could populate my local database with the results. I added the following to my application (cribbed from La.Scala.La):

import scala.dbc._
import scala.dbc.Syntax._
import scala.dbc.syntax.Statement._
import java.net.URI

object PgVendor extends Vendor {
  val uri = new URI("jdbc:postgresql://localhost:5432/dbname")
  val user = "dbuser"
  val pass = "dbpass"

  val retainedConnections = 5
  val nativeDriverClass = Class.forName("org.postgresql.Driver")
  val urlProtocolString = "jdbc:postgresql:"
}

object Runner extends Application {
  val db = new Database(PgVendor)

  val rows = db.executeStatement {
    select fields ("code" of characterVarying(20)) from ("site") where ("responsible_profile_id = 4")
  }
  for (val r <- rows;
       val f <- r.fields) {
    try {
      val str = f.content.sqlString
      val Station = new UsgsWaterQualityStationParser(str)
      println(Station.getMetaData)
    }
  }

  db.close
}

The documentation on the scala.dbc library is woefully lacking, but I managed to pick up some information using a scala wiki detailing the seemingly orphaned scala.dbc2 library. It seemed to work, and gave me enough information to build a proper executeStatement.

The problem was that it didn’t work.

After some Hardy Boy-like investigation, I discovered a very interesting problem that I never expected.² It’s related to the sqlString’s return value, and has something to do with how Scala deals with double vs. single quotes.

Line 25 of the code shows the creation of a value using sqlString(), which, according to what documentation I can find, returns a Java String object. Because UsgsWaterQualityStationParser takes a String in the constructor, everything should work out. The problem, the string returned from sqlString() contains single quotes. Thus, the URL that gets sent to the USGS website is something like:

http://qwwebservices.usgs.gov/Station/search?siteid=USGS-'12345678'

instead of what it should properly be, which is:

http://qwwebservices.usgs.gov/Station/search?siteid=USGS-12345678

It’s a bit strange to me that sqlString would return a String value containing quotes, but I expect it’s because of something that I’m misunderstanding about the scala.dbc library.

As a bit of a hack, I just used String’s replace method:

    try {
      val str = f.content.sqlString.replace("'","")
      val Station = new UsgsWaterQualityStationParser(str)
      println(Station.getMetaData)
    }

Which worked. Running this³ I was able to print out the metadata for the sites that I have in our database that are under the responsibility of the USGS. This solved the problem of getting the data, even if it left open the mystery of the quotes for a later time.

All that’s left now is to create the query such that only sites without metadata will be used (allowing me to do, say 50 sites at a time to limit server requests), and create insert statements to load the metadata into my database. The result, all the information I could want about each USGS site, in the USGS’s own format.

Coda

Despite the learning curve, I’m still having fun with Scala. The issues I have with type safety (I’m a Pythonic Duck Typer) and trying to wrap my head around Scala’s syntax and Java connectivity are a small price to pay for the power, ease and flexibility of the language.

1. See the original post for the code
2. That’s a great euphemism for “I screwed up and created a bug”
3. Not in full, since I didn’t want to create a DOS attack on the USGS’s REST server with 21,000 requests!

Lately, many of my projects– including my really really big one– have been in Java. Since I haven’t programmed in Java since about 1998 (about when I picked up Python, notably) it’s been a hard road. Java has become a harsh mistress. That sweet young thing that was so easy going and flexible so many years ago has grown up to be a cynical, hard-edged woman with a riding crop in her hand.

At least, that’s been my recent experience.

Still, it’s not all been bad. One thing that’s great about it is that I’ve discovered the Scala programming language.

If Python and Java got together and had a son, and then if Haskell and Ruby got together and had a daughter, and then that son and daughter got together and had a baby, that baby would be named Scala.

Scala is a scripting language for Java that is powerful, yet fast. It’s super OOP-centric, yet still has a foot planted firmly in functional programming land. Simple in its syntax, yet able to use any Java classes natively, and it byte compiles to native Java code.

Scala is everything I remember Java promising back in the mid-90s. It’s basically a language that a Java-whipped Python programmer could only dream of, yet it’s real.

Still, as when learning any language, there are times when I smack my head. Today’s head smacker illustrates why it’s important to not make assumptions when programming in an unfamiliar language.

My XML Parser

Today, I was building an XML parser that would grab water quality station information from the websites of organizations like the Army Corps and USGS. This is normally something that I’d do in Python– I’d whip it out real fast and then forget about it– but I thought this would be another good chance to play with Scala.

It’s a very simple language,¹ my HTTP request client consists of only the following:²

import java.io.InputStream;
import java.net.URL;

object Http {
  def request(urlString:String): (Boolean, InputStream) =
    try {
      val url = new URL(urlString)
      val body = url.openStream
      (true, body)
    }
    catch {
      case ex:Exception => (false, null)
   }
}

That’s all that’s needed to make the HTTP request to a server. As you can see, we can pull in the Java classes for InputStream and URL and use them natively. That’s quite nice.

Scala’s design is a bit strange to me. The fact that you can’t have static methods in a class (they go in objects) is a little head scratching sometimes. Also, if you have both static and instance methods, and you do so by naming both a class and an object with the same name. Furthermore, you can have traits, which are somewhat like Java interfaces. Thus, you could have three completely separate types with the same name, all which function both independently and together. Talk about head scratching.

While trying to get used to all this, I defined a base parser as a trait, because I’ll likely be creating parsers for a lot of different types of data sites.

trait BaseParser {
  def baseUrl:String = "http://"
  def fetchAndParseURL(URL:String) {
      val (true, body) = Http request(URL)
      val xml = XML.load(body)
      xml
   }
  def fetchAndParseQuery(query:String) = fetchAndParseURL(baseUrl + query)
}

Here is the skeleton of basic functionality for actually grabbing and parsing the XML of a website.³ fetchAnParseURL() is the basic method, which grabs a URL and parses the XML (error checking and unit tests stripped here). fetchAndParseQuery() is a way to generically modify the base url, with specific modifications to be made in the class.

There are some interesting things in how Scala defines methods. The biggest one is the lack of return statements. Scala assumes that the last object in a method returns. That’s a bit like magic, sometimes, and thus somewhat scary. Another one is the tendency to use equal signs and not require parentheses (as in baseUrl, which is a method). Another nice thing to note here is that function definitions can be declared on one line. It makes some class definitions quite small:

class USGSStation(siteID:String) extends Application with BaseParser {
  override def baseUrl = "http://qwwebservices.usgs.gov/Station/search?siteid=USGS-"
  def GetMetaData() { this.fetchAndParseURL(baseUrl + siteID) }
}

So, here’s my first parser class. It takes a USGS site ID as a string, and overrides with a baseUrl that coincides with the “station identification” REST query. It’s as simple and fast as Python, which blows my mind.

But it doesn’t work.

Smacking Your Head With A Unit

So, I spent a long time trying to figure out why this wouldn’t work. I mean, it compiled, and it ran, and it returned something. It just didn’t return what I wanted. I kept getting a null value of the type “Unit.”

Well, it turns out that this is the “don’t make assumptions” part of learning a new language. You see, in every other language I know except JavaScript which was designed explicitly to torture terrorist suspect detainees, methods are defined with a signature followed by the definition. That’s it. Signature, definition, done.

Scala, is different. I thought that the equal sign was a clever way of making one line functions. No, it turns out that it’s a necessary part of defining a method– at least if you want it to return anything.

Strangely, Scala doesn’t break when you mis-define a method, it just returns Unit– which, as much as I can tell, is the number 42. Once I figured out that my methods needed equal signs, everything worked as expected.

Thus, here is the final XML grabber for the site metadata:

import xml.{Elem, XML}
import java.io.InputStream;
import java.net.URL;

object Http {
   def request(urlString:String): (Boolean, InputStream) =
      try {
         val url = new URL(urlString)
         val body = url.openStream
         (true, body)
      }
      catch {
         case ex:Exception => (false, null)
      }
}

trait BaseParser {
  def baseUrl:String = "http://"
  def fetchAndParseURL(URL:String) = {
      val (true, body) = Http request(URL)
      val xml = XML.load(body)
      xml
   }
  def fetchAndParseQuery(query:String) = fetchAndParseURL(baseUrl + query)
}

class USGSStation(siteID:String) extends Application with BaseParser {
  override def baseUrl = "http://qwwebservices.usgs.gov/Station/search?siteid=USGS-"
  def GetMetaData() = this.fetchAndParseURL(baseUrl + siteID)
}

…which works correctly. It grabs the site’s metadata as the full XML object, which I can then parse for elements such as county code, name, latitude/longitude and sensor type. I’m using this, in conjunction with a bunch of other as-yet-poorly-written-code to pull both sites and site data from their pseudo-REST interface.

Coda

So far, I’m really impressed. Scala is a great language. The total time I spent on this (without including my stupid lack of equal sign problem) was not much more than what I’d spend on a Python version, and that’s without me being familiar with the language. More than that, we can byte compile it and use it within our larger infrastructure without resorting to something like Jython or another solution.

1. and this example was made simpler by the fact that Raphael Ferreira already built up the code to parse Amazon’s website
2. Code here, except where otherwise licensed, is licensed under the MIT license.
3. yes, fellow geeks, I know that there are things I could do better here. This is a quick skeleton to get used to the language, not a final product that needs to be judged. Save that for when I can actually make something work