Groovy could be used as the configuration file for Jenkins workflow. Although I totally don’t make head or tail of Groovy, its syntax is not hard to learn.
How to iteratea list
To export a bunch of tables to CSV format, we could use
[
"table1",
"table2",
"table3",
].each {
table_name ->
sh "export ${table_name} to ${table_name}.csv"
}
Get the output of the shell
We could run shell commands in the Groovy file and then get its output.
all_files = sh (
script: 'ls -lh',
returnStdout: true
).trim()
echo "All files: ${all_files}"
Recently I dug out my USBasp tool and a few AVR microcontrollers, for enjoying programming the C language again. Unexpectedly, the old ATTINY2313V and ATmega88V couldn’t work with my USBasp tool (maybe they have already been fused with an external crystal but I don’t have one at hand). The only two pieces that could work are ATmega16A and ATmega16L. At least I could still have some fun with it.
Later when glancing over the documents of Atmel’s new ATTINY series, I found out that the ATTINY13A only have 1KB space to store program. My example of waterfall light was compiled out to a 2KB hex file. Does that mean I couldn’t put my program into the ATTINY13A? How could I get the real occupation size of flash for the hex file?
Here is one solution by using avr-size
$ avr-size main.hex
text data bss dec hex filename
0 756 0 756 2f4 main.hex
Only 756 bytes (text + data) will be used in flash, so the ATTINY13A should be okay.
Now, what should I do if I want to reduce the size of the binary file compiled from my code? Here is the guide from Atmel.
First, I change the type of the variable “mode” from “unsigned int” to “unsigned char”, this leads the binary file to 726 bytes. Then change all inner functions to “static” (I guess this removed some unused symbol for external linking), reduce the binary size to 668 bytes.
—— 2021.07.13 ——
Furthermore, when I use “-mrelax” option in gcc-avr for linker relaxation, the binary size shrink to 656 bytes.
As the above menu show in the Vertex AI, it is trying to include all common processes of building and running a machine learning model.
For my experiment, I just create a Dataset by loading file from GCS. Unfortunately, the loading process support only CSV file as tabular data so I have to convert my big PARQUET file into CSV format first (really inconvenient).
Strange error
But after I created a training process by using builtin XGBoost container. It report a strange error:
There is an invalid column, but what’s the name of it? The GUI didn’t show. I finally find out that it’s a column with an empty name. Seems Vertex AI couldn’t even process a table with a column of an empty name.
2. AutoML
After manually removed the column with an empty name and select AutoML for my tabular data. The training went successfully. The final regression L1 loss is 0.237, just the same result with my own LightGBM model.
3. Custom Pakcage
By following this document, I create a custom Python package for my training of the XGBoost model. The self-brew package use environment-variable to get Dataset from GCS. The final L1 loss is slightly worse than LightGBM.
Frankly speaking, I haven’t seen any advantage of Vertex AI over our home-brew Argo/K8S training framework. In the Vertex AI training process, those special errors, like OOM(Out Of Memory), are hard to discover.
Normally, to upgrade a cluster of Google Kubernetes Engine, we need to upgrade the master at first, and then node_pools. For convenience, I just click the button “UPGRADE AVAILABLE” in the “Release Channel” section under the “DETAILS” tab of the cluster GUI.
After about 5-10 mins, I started to use command to upgrade all our node_pools in this cluster
for pool in highcpu-2 highcpu-4 highcpu-8 ; do
echo y|gcloud container clusters upgrade my-cluster --node-pool="${pool}" --zone=my-zone --project=my-project
done
Since we have quite a bunch of node_pools, this upgrade process takes about half an hour to finish. Even the script has ended, there was still two node_pools showed “Error” status. And the detail of the error is:
Insufficient quota to satisfy the request: waiting on IG: instance https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a/instances/my-zone-highcpu-2-e435c7e2-xelk is still CREATING. Last attempt error: [QUOTA_EXCEEDED] Instance ‘my-zone-highcpu-2-e435c7e2-xelk’ creation failed: Quota ‘CPUS’ exceeded. Limit: 1200.0 in region …
Seems the upgrade process need to create extra new nodes with a new version first and then delete the old nodes hence would require extra CPUs and cause the exceeding of CPU quota.
Don’t worry. We just need to rerun the upgrade for these two node_pools with “Error” status and eventually all cluster upgraded to 1.19.9-gke.1900
If you accidentally truncate a table in BigQuery, you can try this article to recover the data. Furthermore, I found out that the "bq cp project:dataset.table@-36000 project:dataset.table” method could not work in my situation. The only working solution is “SYSTEM_TIME AS OF“:
CREATE `mydataset.newtable` AS
SELECT *
FROM `mydataset.mytable`
FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR);
and then “bq cp project:mydataset.newtable project:mydataset.mytable“
import numpy as np
np.random.seed(202105)
rand = np.random.rand()
# business logic code using 'rand'
Then I add another np.random.rand() in the head of the code, and this time the output data of this code became quite different.
The reason is simple: the “rand” always generates 0.9 in the previous execution, but since it becomes the second rand() call instead of the first one, it generates 0.06 (unfortunately). The 0.06 is far less than 0.9 so the output data is totally different.
I think the solution is also simple: don’t make your program too dependent on the random number, or just keep your generated number under a range (like 0.6~0.9 in my case).
The program which used CUDA for computing in GPU reported error about memory:
terminate called after throwing an instance of 'std::runtime_error'
what(): [CUDA] an illegal memory access was encountered LightGBM/src/treelearner/cuda_tree_learner.cpp 239
For common C++ program, we use gdb for debugging. For CUDA program, we should use cuda-gdb. Make sure to compile CUDA code with -g flag and then run:
/usr/local/cuda-11.0/bin/cuda-gdb python3
(cuda-gdb) run test.py
After a while, we could see the exact memory corrupt position of the code:
CUDA Exception: Warp Illegal Address
The exception was triggered at PC 0x1668b2f0 (histogram_16_64_256.cu:182)
Thread 1 "python3" received signal CUDA_EXCEPTION_14, Warp Illegal Address.
[Switching focus to CUDA kernel 0, grid 10, block (2163,0,0), thread (0,0,0), device 0, sm 0, warp 3, lane 0]
0x000000001668b380 in LightGBM::histogram16<<<(7360,1,1),(16,1,1)>>> () at LightGBM/src/treelearner/kernels/histogram_16_64_256.cu:185
185 feature = (feature >> ((ind & 1) << 2)) & 0xf;
I have just finished a work about migrating Spark job to BigQuery, or more precisely: migrate Python code to SQL. It’s a tedious work but improve the performance significantly: from 4 hours runtime of PySpark to half an hour on BigQuery (Honors belongs to the BigQuery!).
There are a few notes for the migration, or just SQL skills:
To create or overwrite a temporary table:
CREATE OR REPLACE TEMP TABLE `my_temp_tbl` AS ...
2. Select all columns from a table except some special ones:
4. Using clause OFFSET with LIMIT is terribly slow when the table is very big. The best solution for me is that use “bq extract” to export data to GCS as parquet files, and then get each part of these files by a program.
5. The parquet files could use column names that contain a hyphen, like “last-year”, “real-name”. But the BigQuery only support columns with underline, like “last_year”, “real_name”. So the “bq load” will automatically transfer column name “last-year” in the parquet file to “last_year” in the table of BigQuery.
