All about technology
When using pylint to check my code, it reported an error:
E1137: 'df' does not support item assignment (unsupported-assignment-operation)
for the origin code:
df["column1"] = "hello"
It looks like we have no choice but to use operation .loc in pandas to avoid this error.
# Set all rows in "column1" to "hello" df.loc[:, "column1"] = "hello"
I used awk for quite a long time, but not his brother sed. A couple of days ago I want to insert two lines for a CMake file in a specific position and find a perfect answer: here.
Now I could add two lines by using:
sed -i '/^enable_language/i set(CMAKE_CUDA_ARCHITECTURES 86)\nset(CMAKE_CUDA_COMPILER /usr/local/cuda/bin/nvcc)' \ cmake/public/cuda.cmake
The CMake file changed from:
if("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_C_COMPILER}")
endif()
enable_language(CUDA)
set(CMAKE_CUDA_STANDARD ${CMAKE_CXX_STANDARD})
set(CMAKE_CUDA_STANDARD_REQUIRED ON)to
if("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
set(CMAKE_CUDA_HOST_COMPILER "${CMAKE_C_COMPILER}")
endif()
set(CMAKE_CUDA_ARCHITECTURES 86)
set(CMAKE_CUDA_COMPILER /usr/local/cuda/bin/nvcc)
enable_language(CUDA)
set(CMAKE_CUDA_STANDARD ${CMAKE_CXX_STANDARD})
set(CMAKE_CUDA_STANDARD_REQUIRED ON)I am using Nvidia’s official docker image of PyTorch for my model training for quite a long time. It works very well but the only problem is that the image is too large: more than 6GB. In my poor home network, it would cost a painfully long time to download.
Yesterday, an interesting idea jumped out to my mind: why not build my own small docker image to use PyTorch? Then I started to do it.
Firstly, I chose ClearLinux from Intel since it is very clean and used all state-of-art software (which means its performance is fabulous).
I used distrobox to create my environment:
distrobox create --image clearlinux:latest \
--name robin_clear \
--home /home/robin/clearlinux \
--additional-flags "--shm-size=4g" \
--additional-flags "--gpus all" \
--additional-flags "--device=/dev/nvidiactl" \
--additional-flags "--device=/dev/nvidia0"Enter the environment:
distrobox enter robin_clear
Download CUDA-11.03 run file and install it in the robin_clear:
sudo swupd bundle-add libxml2
sudo ./cuda_11.3.0_465.19.01_linux.run \
--toolkit \
--no-man-page \
--override \
--silentThen, the important part: install gcc-10 (ClearLinux included gcc-12, which is too high for CUDA-11.03) and create the symbol links for it
sudo swupd bundle-add c-extras-gcc10 sudo ln -s /usr/bin/gcc-10 /usr/local/cuda/bin/gcc sudo ln -s /usr/bin/g++-10 /usr/local/cuda/bin/g++
Install the PyTorch:
sudo swupd bundle-add python3-basic pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
Install the apex for mixed-precision training (because my model training is using it):
git clone https://github.com/NVIDIA/apex cd apex pip3 install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Now I can run my training in the ClearLinux. The comparison of these two docker images is here:
| CUDA Version | PyTorch Version | Docker image size | VRAM Usage | Time for training one batch | |
| Nvidia Official PyTorch Image | 11.7 | 1.12.0 | 14.7 GB | 10620MB | 0.2745 seconds |
| My ClearLinux Image | 11.3 | 1.11.0 | 12.8 GB | 10936MB | 0.3066 seconds |
| My ClearLinux Image(v2) | 11.3 | 1.12.0 | 12.8 GB | 10964MB | 0.2812 seconds |
| My ClearLinux Image(build PyTorch myself) | 11.7 | 1.13.0 | 12.8 GB | 10658MB | 0.2716 seconds |
Looks like Nvidia works better than me 🙂 The only chance to win it is by using the newest CUDA and building state-of-art PyTorch manually.
I need to download some large files from Google Drive on my server (“server” means no GUI). After a quick search, I got a solution: https://stackoverflow.com/a/50670037/5048046
We can just install it by using pip:
python3 -m pip install gdown
Then just give the URL of the Google Drive file to it:
# https://drive.google.com/uc?id=<file_id> gdown <file_id>
I am doing some research on TTS (Text-To-Speech) recently and noticed three almost state-of-the-art and also out-of-the-box solutions: LightSpeech (from Microsoft), FastSpeech2 (partly from Microsoft), Nemo (from Nvidia).
The testing text is a paragraph:
The Home Depot, Inc. is the world’s largest home improvement retailer based on net sales for fiscal 2021. We offer our customers a wide assortment of building materials, home improvement products, lawn and garden products, décor products, and facilities maintenance, repair and operations products and provide a number of services, including home improvement installation services and tool and equipment rental. As of the end of fiscal 2021, we operated 2,317 stores located throughout the U.S. (including the Commonwealth of Puerto Rico and the territories of the U.S. Virgin Islands and Guam), Canada, and Mexico. The Home Depot stores average approximately 104,000 square feet of enclosed space, with approximately 24,000 additional square feet of outside garden area. We also maintain a network of distribution and fulfillment centers, as well as a number of e-commerce websites in the U.S., Canada and Mexico. When we refer to “The Home Depot,” the “Company,” “we,” “us” or “our” in this report, we are referring to The Home Depot, Inc. and its consolidated subsidiaries.
The output of FastSpeech2:
it has a lot of noise and sounds like some type of metal.
The output of LightSpeech:
The output of Nemo:
This test is just a summary of my research works and doesn’t mean which algorithm is better than others since the training process will heavily affect the final result. But at least, Nemo is the nearest one to the product scenario.
Apex is a tool to enable mixed-precision training that comes from Nvidia.
import apex.amp as amp net, optimizer = amp.initialize(net, optimizer, opt_level="O2") # forward outputs = net(inputs) loss = criterion(outputs, targets) optimizer.zero_grad() # float16 backward with amp.scale_loss(loss, optimizer) as scaled_loss: scaled_loss.backward() optimizer.step() ... torch.save(net, "model.pth")
After I changed my code to use Apex, it reported an error when saving the model by using torch.save(net, "model.pth")
AttributeError: Can't pickle local object '_initialize.<locals>.patch_forward.<locals>.new_fwd'
Someone has already noticed this problem but it seems no one wants to solve it: link. The only solution for this comes from a Chinese blog: link. It recommends just saving model parameters:
torch.save(net.state_dict(), "model.pth")
When I am trying to log in to my account in a Google Engine VM with “gcloud auth login“, it jumps out a hint:
You are authorizing gcloud CLI without access to a web browser. Please run the following command on a machine with a web browser and copy its output back here. Make sure the installed gcloud version is 372.0.0 or newer. gcloud auth login --remote-bootstrap="https://acounts.google.com/xxxxxxxxxx" Enter the output of the above command:
So I copied this command and ran it on my laptop. But after finishing the process and pressing the “Allow” button, it just jumped to https://cloud.google.com/sdk/auth_success. There is no “output” for me to paste back into the original “cloud auth login“.
Next time if you meet this same situation, please don’t panic. The answer is here: https://stackoverflow.com/a/49885635/5048046
We just need to add the argument “–no-launch-browser”:
gcloud auth login --no-launch-browser
and it will jump out to a web with “putout”
 |
 |
 |
 |
 |
 |
I know it looks too late that I wrote this article in the middle of the year. But, late is much better than none. Right?
I bought the seven books series of “A song of ice and fire” on 23rd November 2019 and finish reading them on 3rd August 2021. It is definitely the best fantasy novel I have read so far (sorry Salvatore, I just told you my true thought). The whole world and the whole story in it are cold and cruel, but unimaginably fascinating and attractive.
The first book of the series is published early in 1996 when I was just a stupid middle school student. I feel really regret I hadn’t started to learn English more intensive and started to read “A song of ice and fire” since then.
The currently last book of the series is published in 2011, a total of eleven years ago. Oh, George, please write the last two volumes as quickly as possible. Your fans had been waiting for 11 years. I don’t want to wait another 11 years more…
Fortunately, I also got some time to look into some area that I am really interesting in but couldn’t benifit my work, such as Semiconductor. The book “Introduction to Semiconductor Technology” really helps me a lot. It starts the introduction from basic phisicals and chemistry knowledge to how to make silicon wafer, to how to paste photoresist, to how to etch, to how lithography, to how to etch, to how to make backend wires. Although forgot almost all my chemistry knowledge before, I still could understand all the process in this industry. And it looks really cool! I can’t hold to show a image for a beautiful DRAM:
In computer science, the algorithm is one of the most important, and also the most difficult part. As a career routine, I found this “The Algorithm Design Manual” and study the algorithms again, mainly focus on the dynamic-programming and just skim through the graph algorithm (they are far away from my daily work, and also far difficult…). I couldn’t say I understood this book very well but at least I take my share of it. And, hope I can revisit it again in the near future.
Because of my wife’s strong recommendation, I started to use Oracle Cloud to host my blog, for free 🙂
Currently, the Oracle Cloud provides four 4-core/24GB ARM64 virtual machines without any fees. This is the best part.
After migrating my blog from AWS (t2.micro) to this ARM64 VM, everything works well except that the permalink of my posts are all changed from date/postname to postnumber. (By following this link and changing my Nginx configuration file /etc/nginx/sites-available/default, this problem has been solved just seconds ago!)
Then I started to think about whether I could deploy other services on this ARM64 VM. Here is my test code for using yolov5:
import cv2
import time
import argparse
import torch
import torch.nn as nn
IMAGE_SHAPE = (300, 300)
def predict(args):
img = cv2.imread(args.image_file)
# detect
model = torch.hub.load('ultralytics/yolov5', 'yolov5m')
net = torch.load(args.classify_model, map_location=torch.device('cpu'))
net.eval()
softmax = nn.Softmax(dim=1)
t0 = time.time()
for _ in range(100):
results = model(img)
for img in results.render():
# Just find the most possible bird (if there are many)
img = cv2.resize(img, IMAGE_SHAPE)
tensor_img = torch.from_numpy(img)
result = net(tensor_img.unsqueeze(0).permute(0, 3, 1, 2).float())
result = softmax(result)
values, indices = torch.topk(result, 10)
t1 = time.time()
print(indices)
print('time:', t1 - t0)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--image_file', default="2400.jpg", type=str, help='Image file to be predicted')
parser.add_argument('--classify_model', default='bird_cls_1160000.pth',
type=str, help='Trained ckpt file path to open')
args = parser.parse_args()
predict(args)It only cost 0.67 seconds for a yolov5m model. This means the Ampere Arm64 core is as fast as AMD EPYC!
I barely pay attention to the pandas.datetime64 type. But yesterday a problem stroke me.
It was a parquet file with a column “start_date”:
>>> df["start_date"]
start_date
0 2022-03-22 00:00:00+11:00
1 2022-03-22 00:00:00+11:00
2 2022-03-22 00:00:00+11:00
3 2022-03-22 00:00:00+11:00
4 2022-03-22 00:00:00+11:00Looks they are “2022-03-22” on Tuesday. But after I export this into BigQuery and select them, they became “2022-03-21 UTC”, which is Monday by default.
The problem is definitely about the Timezone this column has:
>>> df.dtypes start_date datetime64[ns, Australia/Sydney]
What we need to do to be aligned with BigQuery is just remove the timezone and make the time to just “2022-03-22”.
The solution is forcibly and simple:
df["start_date"] = df["start_date"].dt.tz_localize(None)