Scala and the mystery of the quotes

As a kid, I loved the Hardy Boys mysteries, they always had interesting mysteries and plots that were just simple enough that you could figure them out at just the right time. Sometimes, programming is like a mystery, something happens unexpectedly, you hunt down clues, solve the mystery, and find that it leads to deeper mysteries.

A while ago, I started exploring Scala and wrote a post called Head Smacking in Scala on Positively Glorious! (Republished here for convenience). Yesterday I found another head smacking problem that taught me a few things.

Recall that I was creating an XML parser to pull metadata and results from USGS Stream Gauge sites.¹ Yesterday, I decided to explore some of Scala’s DBC abilities so that I could populate my local database with the results. I added the following to my application (cribbed from La.Scala.La):

import scala.dbc._
import scala.dbc.Syntax._
import scala.dbc.syntax.Statement._
import java.net.URI

object PgVendor extends Vendor {
  val uri = new URI("jdbc:postgresql://localhost:5432/dbname")
  val user = "dbuser"
  val pass = "dbpass"

  val retainedConnections = 5
  val nativeDriverClass = Class.forName("org.postgresql.Driver")
  val urlProtocolString = "jdbc:postgresql:"
}

object Runner extends Application {
  val db = new Database(PgVendor)

  val rows = db.executeStatement {
    select fields ("code" of characterVarying(20)) from ("site") where ("responsible_profile_id = 4")
  }
  for (val r <- rows;
       val f <- r.fields) {
    try {
      val str = f.content.sqlString
      val Station = new UsgsWaterQualityStationParser(str)
      println(Station.getMetaData)
    }
  }

  db.close
}

The documentation on the scala.dbc library is woefully lacking, but I managed to pick up some information using a scala wiki detailing the seemingly orphaned scala.dbc2 library. It seemed to work, and gave me enough information to build a proper executeStatement.

The problem was that it didn’t work.

After some Hardy Boy-like investigation, I discovered a very interesting problem that I never expected.² It’s related to the sqlString’s return value, and has something to do with how Scala deals with double vs. single quotes.

Line 25 of the code shows the creation of a value using sqlString(), which, according to what documentation I can find, returns a Java String object. Because UsgsWaterQualityStationParser takes a String in the constructor, everything should work out. The problem, the string returned from sqlString() contains single quotes. Thus, the URL that gets sent to the USGS website is something like:

http://qwwebservices.usgs.gov/Station/search?siteid=USGS-'12345678'

instead of what it should properly be, which is:

http://qwwebservices.usgs.gov/Station/search?siteid=USGS-12345678

It’s a bit strange to me that sqlString would return a String value containing quotes, but I expect it’s because of something that I’m misunderstanding about the scala.dbc library.

As a bit of a hack, I just used String’s replace method:

    try {
      val str = f.content.sqlString.replace("'","")
      val Station = new UsgsWaterQualityStationParser(str)
      println(Station.getMetaData)
    }

Which worked. Running this³ I was able to print out the metadata for the sites that I have in our database that are under the responsibility of the USGS. This solved the problem of getting the data, even if it left open the mystery of the quotes for a later time.

All that’s left now is to create the query such that only sites without metadata will be used (allowing me to do, say 50 sites at a time to limit server requests), and create insert statements to load the metadata into my database. The result, all the information I could want about each USGS site, in the USGS’s own format.

Coda

Despite the learning curve, I’m still having fun with Scala. The issues I have with type safety (I’m a Pythonic Duck Typer) and trying to wrap my head around Scala’s syntax and Java connectivity are a small price to pay for the power, ease and flexibility of the language.

No Comment

1. See the original post for the code [↩]
2. That’s a great euphemism for “I screwed up and created a bug” [↩]
3. Not in full, since I didn’t want to create a DOS attack on the USGS’s REST server with 21,000 requests! [↩]

Comments are closed.