Jul 21, 2019 / EXERCISE, PROGRAMMING, STYLE

Reflecting over Exercises in Programming Style

This week’s post is dedicated to reflection:

In computer science, reflection is the ability of a process to examine, introspect, and modify its own structure and behavior.

This is the 11^th post in the Exercises in Programming Style focus series.Other posts include:

Reflection in Python

The original Python solution uses reflection in different ways:

The exec() function

exec(object[, globals[, locals]])
This function supports dynamic execution of Python code. object must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs). If it is a code object, it is simply executed. In all cases, the code that’s executed is expected to be valid as file input. Be aware that the return and yield statements may not be used outside of function definitions even within the context of code passed to the exec() function. The return value is None.

Simply put, the Python code is written in a string, and then passed to the exec() function to be evaluated e.g.:

extract_words_func = "lambda name : [x.lower() for x in re.split('[^a-zA-Z]+', open(name).read()) if len(x) > 0 and x.lower() not in stops]"
frequencies_func = "lambda wl : frequencies_imp(wl)"
sort_func = "lambda word_freq: sorted(word_freq.items(), key=operator.itemgetter(1), reverse=True)"

exec('extract_words = ' + extract_words_func)  (1)
exec('frequencies = ' + frequencies_func)      (1)
exec('sort = ' + sort_func)                    (1)

1	This defines a new function, which body is in the string above

The locals() function

locals()
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.

Interestingly enough, Python stores functions - and attributes - in a map. Remember that we did that explicitly in one earlier post.

word_freqs = locals()['sort'](locals()['frequencies'](locals()['extract_words'](filename)))

Our first taste of reflection

Using reflection can start quite easily - and we already did that in a previous post - by using function references:

fun stopWords() = read("stop_words.txt")
    .flatMap { it.split(",") }

val funcStopWords = ::stopWords.invoke()

The usage of function references is not similar to the Python sample. As mentioned above, there are two aspects to reflection:

Invocation through reflection
Execution of valid Kotlin code encoded as string

Fortunately, Kotlin allows both. Let’s focus on the first part.

In the initial step, we need to move functions to a dedicated class.

class Reflective {
  fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }
  // Other functions go here
}

In the next step, we shall use Kotlin’s Reflection API. At the center of the API lies the KClass, the equivalent of Java’s Class. Here’s a very simplified class diagram of the API:

The API allows this kind of code:

val clazz: KClass<out Any> = Class.forName("c.f.b.eps.Reflective").kotlin   (1)
val funcStopWords: KFunction<*> = clazz.declaredMemberFunctions.single {
                                    it.name == "stopWords"                  (2)
                                  }
val constructor: KFunction<Any>? = clazz.primaryConstructor                 (3)
val instance: Any? = constructor?.call()                                    (4)
val stopWords: List<String> = funcStopWords.call(instance) as List<String>  (5)

1	To get a `KClass` instance, Kotlin provides the `kotlin` extension property on the `java.lang.Class` class
2	Because functions are stored in a collection and not a map, one needs to filter out those whose name don’t match
3	The same applies to constructors. However, there’s a single primary constructor.
4	Creating a new instance is as simple as calling the constructor
5	Other functions can be called as well: the first object passed as a parameter is the instance the function will be called on

Turning the tables

The above code is great, with one issue IMHO: we actually had to move the functions to a dedicated class. If this wouldn’t have been done, we would have got the following error at compile-time:

Packages and file facades are not yet supported in Kotlin reflection.

Interestingly enough, this is not the case with the Java Reflection API… in Kotlin. Hence, by migrating from Kotlin’s reflection to Java’s reflection, we could put the functions directly at the root! Let’s do that:

fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }

Now, we need to use the exact counterpart of Kotlin’s API:

val clazz: Class<*> = Class.forName("ch.frankel.blog.eps._17_ReflectiveKt")       (1)
val funcStopWords: Method = clazz.getMethod("stopWords")                          (2)
val words: List<String> = funcExtractWords.invoke(null, filename) as List<String> (3)

1	Get a handle on the class. Note that despite being at the root, functions will be generated as static methods of a class anyway. There’s a default class name, but it can be overriden by using the `@JvmName`. It’s not the case here though, therefore the convoluted name
2	The `Class.getMethod()` method is the traditional way to get a reference on a method In Java, methods are held in a map, so it’s possible to get them directly, as opposed to Kotlin
3	Because root functions become static methods, the first parameter to `invoke()` needs to be `null`, as there’s no instance involved

Source code generation and compilation

So far, we called code reflectively. However, we didn’t execute strings as Kotlin code as is the case in Python. In Python - and JavaScript - it’s easy, as those are interpreted languages. In Kotlin, however, source code must first be compiled before being executed.

Aye, there’s the rub: there’s no compilation API in Kotlin at the time of this writing, as opposite to Java (since 1.6). Therefore, it’s not possible - to the extent of my knowledge - to easily compile source code on the fly. An option is to use a build tool, with distinct modules:

one to generate the source code
one to compile it
one to use it

String concatenation is not a really interesting way to generate source code. An alternative is to use a dedicated API for that, namely Kotlin Poet.

KotlinPoet is a Kotlin and Java API for generating .kt source files.

Source file generation can be useful when doing things such as annotation processing or interacting with metadata files (e.g., database schemas, protocol formats). By generating code, you eliminate the need to write boilerplate while also keeping a single source of truth for the metadata.

I believe usage of Kotlin Poet fills the role of Python’s exec().

To use Poet, the architecture can be refined to the following two modules:

Source code generation: This module uses the Kotlin Poet API to generate the source code through a Maven plugin.
Main: This module calls the former plugin to generate the source code into an additional source folder. Generation needs to be done in an early phase of the lifecyle, before compile, e.g. generate-sources. Then, compilation will manage the main source folder, as well as the additional one.

Kotlin Poet could be the theme of several blog posts. It’s centered around two principles, the Builder pattern and a fluent API.

Here’s a simplified class diagram:

Here’s an abridged sample, please check the Git repo for the whole source code:

val packageName = "ch.frankel.blog"                                        (1)
val className = "Reflective"                                               (1)

val stopWordsSpec = FunSpec.builder("stopWords")
  .addCode("""return read("stop_words.txt").flatMap { it.split(",") }""")  (2)
  .build()

val file = FileSpec.builder(packageName, className)
  .addType(
    TypeSpec.classBuilder(ClassName(packageName, className))
      .addFunction(stopWordsSpec)
      .build()
  )
  .build()

file.writeTo(System.out)                                                   (3)

1	Create constants to avoid repeating the string later
2	String blocks allow to use double quotes in the block code without escaping them
3	Dump the generated source code on the standard out. In "real" code, this gets written to a file.

Executing the above code will print the following to the standard out:

package ch.frankel.blog

class Reflective {
    fun stopWords() = read("stop_words.txt").flatMap { it.split(",") }}

Conclusion

Several ways are available to dynamically handle code at runtime. Some languages e.g. C, Scala and Clojure - offer macros. Macros allow some form of pre-processing of source code.

Other languages - e.g. Java and Kotlin (and Scala too) offer reflection. It provides the ability to access and execute code in a dynamic way at runtime. It also allows to generate source code. However, one should be aware that reflection comes at a definite performance cost.

The complete source code for this post can be found on Github.

Follow @nicolas_frankel