/ EXERCISE, PROGRAMMING, STYLE

Reflecting over Exercises in Programming Style

This week’s post is dedicated to reflection:

In computer science, reflection is the ability of a process to examine, introspect, and modify its own structure and behavior.

This is the 11th post in the Exercises in Programming Style focus series. Other posts include:

  1. Introducing Exercises in Programming Style
  2. Exercises in Programming Style, stacking things up
  3. Exercises in Programming Style, Kwisatz Haderach-style
  4. Exercises in Programming Style, recursion
  5. Exercises in Programming Style with higher-order functions
  6. Composing Exercises in Programming Style
  7. Exercises in Programming Style, back to Object-Oriented Programming
  8. Exercises in Programming Style: maps are objects too
  9. Exercises in Programming Style: Event-Driven Programming
  10. Exercises in Programming Style and the Event Bus
  11. Reflecting over Exercises in Programming Style (this post)
  12. Exercises in Aspect-Oriented Programming Style
  13. Exercises in Programming Style: FP & I/O
  14. Exercises in Relational Database Style
  15. Exercises in Programming Style: spreadsheets

Reflection in Python

The original Python solution uses reflection in different ways:

The exec() function
exec(object[, globals[, locals]])

This function supports dynamic execution of Python code. object must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs). If it is a code object, it is simply executed. In all cases, the code that’s executed is expected to be valid as file input. Be aware that the return and yield statements may not be used outside of function definitions even within the context of code passed to the exec() function. The return value is None.

Simply put, the Python code is written in a string, and then passed to the exec() function to be evaluated e.g.:

extract_words_func = "lambda name : [x.lower() for x in re.split('[^a-zA-Z]+', open(name).read()) if len(x) > 0 and x.lower() not in stops]"
frequencies_func = "lambda wl : frequencies_imp(wl)"
sort_func = "lambda word_freq: sorted(word_freq.items(), key=operator.itemgetter(1), reverse=True)"

exec('extract_words = ' + extract_words_func)  (1)
exec('frequencies = ' + frequencies_func)      (1)
exec('sort = ' + sort_func)                    (1)
1This defines a new function, which body is in the string above
The locals() function
locals()

Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.

Interestingly enough, Python stores functions - and attributes - in a map. Remember that we did that explicitly in one earlier post.

word_freqs = locals()['sort'](locals()['frequencies'](locals()['extract_words'](filename)))

Our first taste of reflection

Using reflection can start quite easily - and we already did that in a previous post - by using function references:

fun stopWords() = read("stop_words.txt")
    .flatMap { it.split(",") }

val funcStopWords = ::stopWords.invoke()

The usage of function references is not similar to the Python sample. As mentioned above, there are two aspects to reflection:

  1. Invocation through reflection
  2. Execution of valid Kotlin code encoded as string

Fortunately, Kotlin allows both. Let’s focus on the first part.

In the initial step, we need to move functions to a dedicated class.

class Reflective {
  fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }
  // Other functions go here
}

In the next step, we shall use Kotlin’s Reflection API. At the center of the API lies the KClass, the equivalent of Java’s Class. Here’s a very simplified class diagram of the API:

Reflection API class diagram

The API allows this kind of code:

val clazz: KClass<out Any> = Class.forName("c.f.b.eps.Reflective").kotlin   (1)
val funcStopWords: KFunction<*> = clazz.declaredMemberFunctions.single {
                                    it.name == "stopWords"                  (2)
                                  }
val constructor: KFunction<Any>? = clazz.primaryConstructor                 (3)
val instance: Any? = constructor?.call()                                    (4)
val stopWords: List<String> = funcStopWords.call(instance) as List<String>  (5)
1To get a KClass instance, Kotlin provides the kotlin extension property on the java.lang.Class class
2Because functions are stored in a collection and not a map, one needs to filter out those whose name don’t match
3The same applies to constructors. However, there’s a single primary constructor.
4Creating a new instance is as simple as calling the constructor
5Other functions can be called as well: the first object passed as a parameter is the instance the function will be called on

Turning the tables

The above code is great, with one issue IMHO: we actually had to move the functions to a dedicated class. If this wouldn’t have been done, we would have got the following error at compile-time:

Packages and file facades are not yet supported in Kotlin reflection.

Interestingly enough, this is not the case with the Java Reflection API…​ in Kotlin. Hence, by migrating from Kotlin’s reflection to Java’s reflection, we could put the functions directly at the root! Let’s do that:

fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }

Now, we need to use the exact counterpart of Kotlin’s API:

val clazz: Class<*> = Class.forName("ch.frankel.blog.eps._17_ReflectiveKt")       (1)
val funcStopWords: Method = clazz.getMethod("stopWords")                          (2)
val words: List<String> = funcExtractWords.invoke(null, filename) as List<String> (3)
1Get a handle on the class. Note that despite being at the root, functions will be generated as static methods of a class anyway. There’s a default class name, but it can be overriden by using the @JvmName. It’s not the case here though, therefore the convoluted name
2The Class.getMethod() method is the traditional way to get a reference on a method In Java, methods are held in a map, so it’s possible to get them directly, as opposed to Kotlin
3Because root functions become static methods, the first parameter to invoke() needs to be null, as there’s no instance involved

Source code generation and compilation

So far, we called code reflectively. However, we didn’t execute strings as Kotlin code as is the case in Python. In Python - and JavaScript - it’s easy, as those are interpreted languages. In Kotlin, however, source code must first be compiled before being executed.

Aye, there’s the rub: there’s no compilation API in Kotlin at the time of this writing, as opposite to Java (since 1.6). Therefore, it’s not possible - to the extent of my knowledge - to easily compile source code on the fly. An option is to use a build tool, with distinct modules:

  1. one to generate the source code
  2. one to compile it
  3. one to use it

String concatenation is not a really interesting way to generate source code. An alternative is to use a dedicated API for that, namely Kotlin Poet.

KotlinPoet is a Kotlin and Java API for generating .kt source files.

Source file generation can be useful when doing things such as annotation processing or interacting with metadata files (e.g., database schemas, protocol formats). By generating code, you eliminate the need to write boilerplate while also keeping a single source of truth for the metadata.

I believe usage of Kotlin Poet fills the role of Python’s exec().

To use Poet, the architecture can be refined to the following two modules:

Source code generation

This module uses the Kotlin Poet API to generate the source code through a Maven plugin.

Main

This module calls the former plugin to generate the source code into an additional source folder. Generation needs to be done in an early phase of the lifecyle, before compile, e.g. generate-sources. Then, compilation will manage the main source folder, as well as the additional one.

Kotlin Poet could be the theme of several blog posts. It’s centered around two principles, the Builder pattern and a fluent API.

Here’s a simplified class diagram:

Kotlin Poet API

Here’s an abridged sample, please check the Git repo for the whole source code:

val packageName = "ch.frankel.blog"                                        (1)
val className = "Reflective"                                               (1)

val stopWordsSpec = FunSpec.builder("stopWords")
  .addCode("""return read("stop_words.txt").flatMap { it.split(",") }""")  (2)
  .build()

val file = FileSpec.builder(packageName, className)
  .addType(
    TypeSpec.classBuilder(ClassName(packageName, className))
      .addFunction(stopWordsSpec)
      .build()
  )
  .build()

file.writeTo(System.out)                                                   (3)
1Create constants to avoid repeating the string later
2String blocks allow to use double quotes in the block code without escaping them
3Dump the generated source code on the standard out. In "real" code, this gets written to a file.

Executing the above code will print the following to the standard out:

package ch.frankel.blog

class Reflective {
    fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }}

Conclusion

Several ways are available to dynamically handle code at runtime. Some languages e.g. C, Scala and Clojure - offer macros. Macros allow some form of pre-processing of source code.

Other languages - e.g. Java and Kotlin (and Scala too) offer reflection. It provides the ability to access and execute code in a dynamic way at runtime. It also allows to generate source code. However, one should be aware that reflection comes at a definite performance cost.

The complete source code for this post can be found on Github.
Nicolas Fränkel

Nicolas Fränkel

Nicolas Fränkel is a Developer Advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). Usually working on Java/Java EE and Spring technologies, but with focused interests like Rich Internet Applications, Testing, CI/CD and DevOps. Currently working for Exoscale. Also double as a teacher in universities and higher education schools, a trainer and triples as a book author.

Read More