JSON – Robin on Linux

Using loop in Jsonnet

Jsonnet is a templating language and tool to generate JSON/YAML files. Since already have a language instead of configuration, we can generate a bunch of configuration issues with simple code.

For example, I use loop in Jsonnet to write repeat items:

local food_type = ['cake', 'fruit', 'vegetable', 'rice', 'eggs', 'milk', 'meat', 'pizza'];
local final_meal = [
  { name: 'food_' + type, value: 'like' }
  for type in food_type
];
{ 'sample.yaml': std.manifestYamlDoc(
  final_meal,
) }

The std.manifestYamlDoc is used for generate YAML file instead of JSON.

After run jsonnet sample.jsonnet -m . -S, it will generate sample.yaml as:

- "name": "food_cake"
  "value": "like"
- "name": "food_fruit"
  "value": "like"
- "name": "food_vegetable"
  "value": "like"
- "name": "food_rice"
  "value": "like"
- "name": "food_eggs"
  "value": "like"
- "name": "food_milk"
  "value": "like"
- "name": "food_meat"
  "value": "like"
- "name": "food_pizza"
  "value": "like"

Less code but more human readable. I think this is the power of Jsonnet.

Import data to Redis from JSON format

Redis use rdb file to make data persistent. Yesterday, I used the redis-rdb-tools for dumping data from rdb file to JSON format. After that, I write scala code to read data from JSON file and put it into Redis.
Firstly, I found out that the JSON file is almost 50% bigger than rdb file. After checking the whole JSON file, I make sure that the root cause is not the redundant symbols in JSON such as braces and brackets but the “Unicode transformation” in JSON format, especially the “\u0001” for ASCII of “0x01”. Therefore I write code to replace it:

val key: String = line.stripPrefix("\"").stripSuffix("\"").replaceAll("\\\\u0001", "^A")

Then the size of JSON file became normal.
There was still another problem. To read the JSON file line by line, I use code from http://naildrivin5.com/blog/2010/01/26/reading-a-file-in-scala-ruby-java.html:

import scala.io._
object ReadFile extends Application {
  val s = Source.fromFile("some_file.txt")
  s.getLines.foreach( (line) => {
    println(line.trim.toUpperCase)
  })
}

But this code will load all data from file and then run “foreach”. As my file is bigger than 100GB, it will cost too much time and RAM ….
The best way is to use IOStream in java:

val dataFile = new File("my.txt")
var reader = new BufferedReader(new FileReader(dataFile))
var line: String = reader.readLine()
while (line != null) {
    // do something
    line = reader.readLine()
}

This is exactly read file “line by line”.