The Grey Lens Man's View

Making Scala (The Language) Scalable

2011-06-11T07:16:00.000-07:00

Building Large Systems

Say "Scala" and one receives any number of interesting reactions. From Kurtz, "The Types! The Types!" to Ordersky " ... it's in fact a rather simple language to program in."

Building large systems requires a robust typing language. Yet, typing is not sufficient alone, more is needed.

Undoubtedly Scala's Cosmological Type Horizon will continue to expand indefintely (Λ > 1) and justifiably so. In Scalaland, Typing, yea we got that.

However, it is Scala - The Scalable Language, and not say Typla - The Typeful Language.

"The name Scala stands for “scalable language.” The language is so named because it was designed to grow with the demands of its users. You can apply Scala to a wide range of programming tasks, from writing small scripts to building large systems."

Building large systems - comprehensible, changeable, manageable and maintainable ones that work thank you very much. At the end of the day isn't that what all of this is about?

Scala Bringing A Knife To A Gunfight?

Scala's extant weapons of scale are the package, the class, the object, the trait, and package object. Set the package object aside for the moment and examine the remaining constructs.

If Scala's syntax are the words, declarations and definitions the sentences then traits, classes, and objects are the paragraphs. What is missing are the analogous constructs for chapters, books and dare I say sets and volumes? Packages, imposed upon Scala by Java, offer name scoping and importation, but lack the gravitas to get the job done.

Template nesting allows for vertical structuring. Traits and open/free/abstract declarations allow for horizontal structuring, interfacing and more. Scala has well thought out qualified access modifiers, public, private, protected, sealed.

Very solid feature set for ... Smala, a small and medium scaling language. Best in class.

But All I Want To Do Is X

Parse a bit of XML, create an Actor, a web page, sort and iterate through a simple sequence of instances. What is the problem? Too much information. The exposed surface area far exceeds the general needs of the user of these libraries. I just want to use your really well crafted library, is the endoscopy really necessary?

Ok the examples are admittedly rather poor ones. But the idea conveyed should resonate.

The common theme is a top level package(s), multiple subpackages, traits, types, objects, classes, dozens of them if not a couple hundred which ultimately exposes a rather small, maybe use specific, surface area (API). Probably a small team, or dedicated person will be responsible for delivering one of these structures and its interface. A large system will be composed from a myriad of these pyramidal package structures working together to form a large scale software system.

Packages Are Not Modules

Modularization is the solution to large scale software systems. Modularization, in the sense of information hiding and modularizing around design decisions. For this packages are not sufficient; neither are classes, objects or traits. Import is not inheritance – why we need both: modules and classes.

Scalable Languages Require Modules

If one accepts the argument so far, then a scalable language requires modules, Scala claims to be scalable language from conceptualization, ergo Scala's modules must... wait a minute. Where'd they go?

Exactly the point. Of course the Scala team know all about modules, their need and ways to implement them. It is only a question of perceived necessity and priority of implementation. An interesting exercise is to grep the scala code base and search the language specification for the word "module". Fairly prevalent isn't it?

Maybe they've concluded template nesting with stable and dependent type paths is the solution. No new constructs are required, that all the necessary machinery is in place. But as recently as Scala 2.8 I believe they were hunting bigger game. It's those package objects.

Package Objects

A simplistic view of package objects is to view them as a scoped global (to package) namespace in which one places implicits, vals and types. A powerful concept in and of itself. The scala package object demonstrates things nicely. This is where Lists, allowing List (1,2,3) anywhere, appear as if they were built into the language, not as something built atop the language implemented as just another library (somewhere else).

Vehicles of package convenience yes, but not the foundation for scaling million line software systems. Or are they?

Package Objects As Module Interface(s)

A tantalizing teaser of enhanced package objects can be found on the Scala website. The most interesting part is here.

An interesting question is what should happen when the name of a top-level class or object in a package is redefined in the associated package object. For the moment, this is simply forbidden. But if we wanted to allow this, it would make sense to take the definition in the package object as the public interface of the definition in the package itself. Code from within the package could see the full definition, but code from outside the package could only see what's defined in the package object. That way, package objects could evolve into the backbone of a fairly advanced module system, which allows to flexibly combine interfaces with implementations.

Fully evolved package objects would act as module interface definitions. An exposed surface area of implicits, classes, objects, traits and types.

The corresponding Scaladoc generation is focused on use(rs) via the module interface. The existent package Scaladoc remains comprehensive for library maintainers, extenders and power users. Possibly multiple modules interface definitions such as simple and full.

Consider the collections library with both a simple module interface containing just the top level collection classes, objects and public methods. And a second, full module interface exposing the complete bones and guts for extension and reuse outside the core package(s).

Sealed module interfaces, opaque versus transparent, intra-module requires plus provides declarations. Components? Module initialization code. Such a top level initialization layer could allow for object parameters, objects with constructors, where the static instantiating information is defined in the module interface definition for object singleton instances etc. Scoped custom application specific predefs. Lots of possibilities.

Scala What's Next?

Scala is one of the most advanced languages around with new types as well as the promoted effort on parallel/concurrent libraries/DSLs are certain. Within the year, Scala's nick name will be Prala, the parallel processing language.

But I don't think it can truly call itself Scala - The Scalable Language without the addition of an advanced module system. For me, this would be my #1 feature request for Scala 3.0

Stone Tools and Scala Development - Part II Building

2009-03-21T10:28:00.000-07:00

Maven2+Scala-Maven-Plugin+ScalaTest+Emacs

In this post I'll fly through how I use Maven in combo with Emacs to build Scala programs.

But first, a couple of additional notes on Part 1, where I covered the basics of how I use Emacs to edit Scala source files.

My one only commenter (thanks Normen) seemed to find the blog useful but skipped the ctags setup. I highly recommend that anyone doing a trial of Emacs for Scala development do the ctags setup, build your tag file and then Alt-. over a type in your source. You just can't code without this.

This function and Alt-/ like completion are often used as reasons why people pick the heavier IDE's over Emacs even though they like the raw editing power of Emacs. They often don't know one can achieve decent support for these features in Emacs as well. Admittedly the IDEs are semantic in analysis where Emacs is syntactic. Ctags, completion and yas-snippets get you close to these capabilities and you can flat out edit code as well.

Second, I posted a small patch to the scala-tools list for a very annoying indentation issue which was just applied in the Scala repository. So either grab the patch from Nabble scala-tools or pull the recent Emacs mode code from subversion. Note the first patch post was incorrect and a correct patch was posted later in the chain.

I customized my colors a bit, and for me editing Scala code looks like this. The lower left corner shows the results of compiling my project from inside Emacs via Maven2 and the Scala Maven plugin.

Maven in and of itself is a big topic and I am not a Maven maven by any means. So I'm only covering the specifics of using Maven + Scala + Emacs.

The Setup

First we want to run Maven inside of Emacs to take advantage of the using Emacs to move from compiler errors to the location of the error in the source code. A google search located a small but nice bit of elisp to run maven from inside Emacs. I modified it somewhat for Scala purposes. Here it is in its entirety.

;; Simple but effective Maven2 support.

(require 'compile)

(defvar mvn-command-history nil
"Maven command history variable")

(defvar scala-compile-error-regex
'("^\\[WARNING\\] \\([a-zA-Z0-9/-]+[.]scala\\):\\([0-9]+\\): error:" 1 2 nil 2 nil))

(defun mvn-compile-buffer-name (mode)
"*Maven Compile*")

(set 'scala-compile-error-regex '("^\\[WARNING\\] \\([a-zA-Z0-9/-]+[.]scala\\):\\([0-9]+\\): error:" 1 2 nil 2 nil))

(set 'compilation-buffer-name-function 'mvn-compile-buffer-name)
(set 'compilation-error-regexp-alist (list scala-compile-error-regex))

(set 'compilation-mode-font-lock-keywords
'(("^\\[ERROR\\] BUILD FAILURE"
(0 compilation-error-face))
("^\\[WARNING\\]"
(0 compilation-warning-face))
("^\\(\\[INFO\\]\\)\\([^\n]*\\)"
(1 compilation-info-face)
(2 compilation-line-face))
("^\\[ERROR\\]"
(0 compilation-error-face))
("\\(Failures\\): [1-9][0-9]*,"
(0 compilation-error-face))
("\\(Failures\\): 0"
(0 compilation-info-face))
("\\(Errors\\): [1-9][0-9]*,"
(0 compilation-error-face))
("\\(Errors\\): 0"
(0 compilation-info-face))
("\\(Skipped\\): [1-9][0-9]*,"
(0 compilation-error-face))
("\\(Skipped\\): 0"
(0 compilation-info-face))
("Tests run: [0-9]+"
(0 compilation-warning-face))
("T E S T S"
(0 compilation-warning-face))
("Compilation finished at [^\n]+"
(0 compilation-info-face))
("^Compilation \\(exited abnormally\\|interrupt\\|killed\\|terminated\\|segmentation fault\\)\\(?:.*with code \\([0-9]+\\)\\)?.*"
(0 '(face nil message nil help-echo nil mouse-face nil) t)
(1 compilation-error-face)
(2 compilation-error-face nil t))))

(defun mvn-at-root (path)
"Determine if the given path is root."
(equal path (mvn-parent-path path)))

(defun mvn-parent-path (path)
"The parent path for the given path."
(file-truename (concat path "/..")))

(defun mvn-pom-at-path-p (path)
"Does a pom.xml exist in the given path."
(file-exists-p (concat path "/pom.xml")))

(defun mvn-find-path-to-pom ()
"Move up the directory tree for the current buffer until root or a pom.xml is found."
(let ((fn (buffer-file-name)))
(let ((path (file-name-directory fn)))
(while (and (not (mvn-pom-at-path-p path))
(not (mvn-at-root path)))
(setf path (file-truename (mvn-parent-path path))))
path)))

(defun mvn-master (&optional args)
(interactive)
"Determine the potential master pom by checking if there is a pom in a pom's parent directory. nil if no parent pom else the path to the parent."
(let ((pom-path (mvn-find-path-to-pom)))
(let ((parent-pom-path (file-truename (concat pom-path "/.."))))
(if (not (mvn-pom-at-path-p parent-pom-path))
(message "No master pom found.")
(compile (read-from-minibuffer "Command: "
(concat "mvn -o -f " parent-pom-path "/pom.xml clean install")
nil nil 'mvn-command-history))))))

(defun mvn (cmd)
"Runs maven in the current project. Starting at the directoy where the file being vsisited resides, a search is
made for pom.xml recsurively. A maven command is made from the first directory where the pom.xml file is found is then displayed
in the minibuffer. The command can be edited as needed and then executed. Errors are navigate to as in any other compile mode"
(interactive "n0-clean 1-install 2-install,bundle:install 3-compile 4-test : ")
(let ((path (mvn-find-path-to-pom)))
(let ((goal (case cmd
((0) "clean")
((1) "install")
((2) "install bundle:install")
((3) "compile")
((4) "test"))))
(if (not (file-exists-p (concat path "/pom.xml")))
(message "No pom.xml found")
(compile (read-from-minibuffer "Command: "
(concat "mvn -o -f " path (concat "/pom.xml " goal))
nil nil 'mvn-command-history))))))

;; String pattern for locating errors in maven output. This assumes a Windows drive letter at the beginning
;;(add-to-list
;; 'compilation-error-regexp-alist
;; '("^\\([a-zA-Z]:.*\\):\\[\\([0-9]+\\),\\([0-9]+\\)\\]" 1 2 3))

First, the color coding stuff is sort of nice. It colorizes compile and unit test errors to red and passed tests to green, informational output to yellow/orange and that sort of thing. It also defines regular expressions which identify scala compiler errors and extraction of the file and line numbers for Emacs' compile.el.

Cut and paste the above in a file mvn.el, put it in your site-lisp directory and modify your .emacs file as follows. Here I repeat my Scala-mode setup as well to show where we bind the F6 key to a function which runs Maven from Emacs.

;;;;;;;;;;;;
;; MAVEN 2
;;;;;;;;;;;;

(let ((path "/usr/local/share/emacs/site-lisp/mvn"))
(setq load-path (cons path load-path))
(load "mvn.el"))

;;;;;;;;;;;;
;; SCALA
;;;;;;;;;;;

(let ((path "/usr/local/share/emacs/site-lisp/scala"))
(setq load-path (cons path load-path))
(load "scala-mode-auto.el"))

(defun scala-turnoff-indent-tabs-mode ()
(setq indent-tabs-mode nil))

;; scala mode hooks
(add-hook 'scala-mode-hook 'scala-turnoff-indent-tabs-mode)

;; Maven bound to
(add-hook 'scala-mode-hook
'(lambda ()
(define-key scala-mode-map '[f6] 'mvn)))

Using mvn.el

As I've indicated, this post is not about using Maven2 and the Scala plugin which is covered quite well in a number of places. See the scala-tools site on how to setup a Maven+Scala project. My focus is using a Maven+Scala from within Emacs and that is done easily enough.

So at this point I am assuming that you have a working pom.xml, and in the standard Maven file layout have one or more Scala source files and if you issue a "mvn install" command from the command line your code builds just fine. To achieve this level of enlightenment I again refer you to the scala-tools site. In addition, I'm assuming you've installed the mvn.el file and setup your .emacs as discussed.

So now lets switch from running Maven from a command line to within Emacs.

Fire up Emacs and open a Scala source file. You have to be in active buffer which contains a Scala source file. Press F6. You should see a small list of Maven goals in the interactive buffer. Pick a number and hit return. And now you have a small Maven command. You can change it by editing it or just hit return to run it.

A *Maven-Compile* buffer should appear within which scrolling Maven output is happening. A typical run for me (notice the complete lack of errors :) ) looks like this.

OK I know what you're thinking. Pretty colors, but what's the big deal. Well first the mvn.el script uses Emacs' compile.el so we can leverage pre-existing functionality to navigate to the compile errors.

Click on an error and you jump to it the source file. Alternatively you can move the cursor over the error msg and hit return. ctrl-x ` navigates to the next error. All this and more is available and explained in detail here.

mvn.el Insight

A couple of notes on mvn.el. Its not very complex and gains much of its power from Emacs compile.el for error navigation.

So what is really happening when the F6 key is pressed? Well first the directory of the Scala source file is determined. Then we look to see if a pom.xml is present, if not we recursively move up the directory tree until one is found or report an error if one is not found. Using the compile command found in compile.el, which does all the heavy lifting, Maven is launched with the located pom.xml and the selected goal.

For "advanced" Maven users who have a top level pom.xml with sub-projects or modules with their own pom.xml there is an interactive command which move up the directory tree and finds the top most pom.xml. alt-x mvn-master

So there it is. The basics of using using Maven+ScalaPlugin+Emacs for Scala program development.

Next post. Scala + OSGi + Maven + Felix + Emacs

Note: I added the "pick a number" thing awhile back, didn't particularly care for it, and never got around to removing it, and got sort of used to it. I just F6 3 RTN without much thinking. But the mvn.el is easily modified. See mvn-master for the old way.

Stone Tools and Scala Development - Part I Editing

2009-02-17T14:10:00.000-08:00

I see great potential in Scala, the programming language.

Scala is a good thing. Scala deserves to win, it deserves to supplant Java in enterprise IT because it is better. If their exists some law of nature dictating that the universe must maintain itself in good kilter and balance every "worse is better" thing with a better thing then Scala will succeed.

Currently the language and compiler exceeds any IDEs capability to properly host it. And a huge proportion of developers have never experienced anything other then the all mothering Eclipse, Netbeans, or Microsoft IDE. Without the skeletal support of the IDE they are rendered near impotent, become jelly like, puddle and quiver back and forth producing nary a line of code.

Thankfully there are Scala plugins for Netbeans, Intellij and Eclipse. Yet there exists no more gnashing teeth on the Scala mailing list then concerning the relative immaturity of these plugins.

Amazingly these days, so interlinked is the concept of computer language and its IDE(s) many no longer see them as less then monolithic. Scala, the language, sucks because the IDE support sucks.

Fortunately, frankly surprisingly, one sees continued steady effort and work invested in improving these plugins. Surprisingly because it is a near thankless job and borders on pure drudgery. We salute you.

Until these tools mature, I claim it is possible to eek out a development existence, even a highly productive one using nothing more then the stone tools of a compiler, editor, make-like system, unix utilities and the command line. Some even view this as the superior, even ultimate IDE.

Here is how I roll with my stone tools for Scala development.

First off Linux. All of details below assume a Linux system.

Editing Scala Code With Emacs.

I know, we all heard it a 1000 times about those emacs gods who transcend the very laws of nature through advanced enlightenment with emacs which simply escapes us low karmic souls.

It's true, if you invest in emacs long enough it starts to pay off like a rigged slot machine. Put you have to feed it a lot of quarters first. I'm far from an emacs guru, but I've reached a point down the 7-fold path where I can't live without it and marvel how I ever did.

Get the latest emacs. Emacs was moribund for years but has experience a renaissance recently. The vastly improved font handling is worth it alone. It is once again better then XEmacs IMHO. Get good ol' emacs, I build it from cvs and it's easy. Standard configure, make, dance and if I remember the README gives an option to bypass configure with the proper Make incantation.

Here is my base .emacs stuff.

(global-font-lock-mode t)

(windmove-default-keybindings)

(setq use-file-dialog nil)

;; "y or n" instead of "yes or no"
(fset 'yes-or-no-p 'y-or-n-p)

;; Highlight regions and add special behaviors to regions.
;; "C-h d transient" for more info
(setq transient-mark-mode t)

;; Display line and column numbers
(setq line-number-mode t)
(setq column-number-mode t)

(global-set-key "\C-c\C-w" 'backward-kill-word)

;; Remove unnecessary gui stuff
(if (fboundp 'scroll-bar-mode) (scroll-bar-mode -1))
(if (fboundp 'tool-bar-mode) (tool-bar-mode -1))
(if (fboundp 'menu-bar-mode) (menu-bar-mode 1))

(blink-cursor-mode 0) ;; no blink

(set-background-color "black")
(set-face-background 'default "black")
(set-face-background 'region "black")
(set-face-foreground 'default "white")
(set-face-foreground 'region "gray60")
(set-foreground-color "white")
(set-cursor-color "red")

This will give you a clean, black background based emacs with maximum screen space, but leaving the main menu, which I found on someones (thank you) emacs blog.

Next install the Scala folks provided scala-mode for emacs. They did a nice job.

svn co http://lampsvn.epfl.ch/svn-repos/scala/scala-tool-support/trunk/src/emacs

Follow the readme and install. The yasnippit stuff is not bad, but I don't use it extensively, but give it a whirl and decide yourself.

For the record here is what my .emacs file looks like after I installed the provided scala-mode and yasnippit for scala.

;;;;;;;;;;;;
;; SCALA
;;;;;;;;;;;

(let ((path "/usr/local/share/emacs/site-lisp/scala"))
(setq load-path (cons path load-path))
(load "scala-mode-auto.el"))

(defun scala-turnoff-indent-tabs-mode ()
(setq indent-tabs-mode nil))

;; scala mode hooks
(add-hook 'scala-mode-hook 'scala-turnoff-indent-tabs-mode)

;; Maven bound to
(add-hook 'scala-mode-hook
'(lambda ()
(define-key scala-mode-map '[f6] 'mvn)))

;;;;;;;;;;;;;
;; YASNIPPIT
;;;;;;;;;;;;;

(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/yasnippet-0.5.7")
(require 'yasnippet)
(yas/initialize)
(yas/load-directory "/usr/local/share/emacs/site-lisp/yasnippet-0.5.7")

More on that Maven key binding later.

Now we can edit Scala code effectively. There are some indentation issues and I've been able to improve it a bit but I still have fix a few things which I broke in the process before it is worthy.

With Yasnippit and alt-/ completion its not too bad. The next piece is exuberant ctags so you can alt-. and jump directly to that class, object or trait definition.

First there is ctags and then there is ctags. Emacs comes with a program called ctags. ctags will index your source code symbols so you can hop around your source code files.

There is a variation of ctags called exuberant ctags found at http://ctags.sourceforge.net/ and you need to install it. Again it should be quick and easy if you've ever done the ol' Linux Make dance.

Be careful, when you make install emacs as it reinstalls its version of ctags so you need to make install exhuberant ctags. Just make sure however you go about it, exhuberant ctags is what is executed in your PATH and not the standard emacs ctags.

In your home directory create a .ctags file with the following.

--langdef=Scala
--langmap=Scala:.scala
--regex-Scala=/^[^\*\/]*class[ \t]*([a-zA-Z0-9_]+)/\1/c,classes/
--regex-Scala=/^[^\*\/]*object[ \t]*([a-zA-Z0-9_]+)/\1/o,objects/
--regex-scala=/^[^\*\/]*trait[ \t]*([a-zA-Z0-9_]+)/\1/t,traits/
--regex-Scala=/^[^\*\/]*case[ \t]*class[ \t]*([a-zA-Z0-9_]+)/\1/m,case-classes/
--regex-Scala=/^[^\*\/]*abstract[ \t]*class[ \t]*([a-zA-Z0-9_]+)/\1/a,abstract-classes/
#--regex-Scala=/^[^\*\/]*def[ \t]*([a-zA-Z0-9_]+)[ \t]*.*[:=]/\1/f,functions/
#--regex-Scala=/^[^\*\/]*val[ \t]*([a-zA-Z0-9_]+)[ \t]*[:=]/\1/V,values/
#--regex-Scala=/^[^\*\/]*var[ \t]*([a-zA-Z0-9_]+)[ \t]*[:=]/\1/v,variables/
#--regex-Scala=/^[^\*\/]*type[ \t]*([a-zA-Z0-9_]+)[ \t]*[\[<>=]/\1/T,types/

I just like to index classes, traits, objects and case-classes. Hmmm just noticed I should add case objects as well.

Now when editing a scala file in emacs, under the emacs Scala menu item, Features you can build and use tag files which index key symbols in your source code.

Explore your Scala editing environment. No two people use emacs in the same fashion.

alt-x rgrep in emacs to ad-hoc search your scala files.
use alt-/ for simple minded completion.
Hit the F1 key to pop up your speed bar window. Note at the bottom you can click and switch to view, buffers, files etc in the speed bar.
alt-x erc and join #scala in your emacs for instant Scala support. :)
Are you using ctr-x o to move between buffers? There are tons of better ways. Open up a several scala buffer windows ctr-x 2 or 3, now hold down shift and use your arrow keys to move around.
Notice that there are 5-6 Tag search options under the Scala Menu, try them all.
While editing a scala file, put your cursor over a type definition (x: MyType, ...) and do alt-.
Toggle yas/snippit, alt-x yas/minor-mode. In your scala source file type 'def ' or 'class ' and hit the tab key.
Run the scala interpreter in emacs its right there in the Scala menu.

Next post I'll address setting up and building your Scala project using Maven from inside emacs.