Finally I get a few hours to look into the code of LightGBM.

I used to have some questions about LighGBM, and now fortunately I can answer some of them by myself. Even some answers may be wrong, that is still better than no answer at all 🙂

Q: Will LightGBM contruct a couple of trees as one model?

A: No. It will only contruct one tree as a model for a dataset

Q: How would LightGBM choose the feature that has the highest gain in entropy?

A: It will simply iterate all features (with a loop, in code) and try to find the best split for all of them. After that, it will pick the feature and the split with the highest gain

"src/treelearner/serial_tree_learner.cpp"
158 Tree* SerialTreeLearner::Train(const score_t* gradients, const score_t *hessians) {
...
185   int init_splits = ForceSplits(tree_ptr, &left_leaf, &right_leaf, &cur_depth);
186
187   for (int split = init_splits; split < config_->num_leaves - 1; ++split) {
188     // some initial works before finding best split
189     if (BeforeFindBestSplit(tree_ptr, left_leaf, right_leaf)) {
190       // find best threshold for every feature
191       FindBestSplits(tree_ptr);
192     }
...
"src/treelearner/serial_tree_learner.cpp"
322 void SerialTreeLearner::FindBestSplits(const Tree* tree) {
323   std::vector<int8_t> is_feature_used(num_features_, 0);
324   #pragma omp parallel for schedule(static, 256) if (num_features_ >= 512)
325   for (int feature_index = 0; feature_index < num_features_; ++feature_index) {
326     if (!col_sampler_.is_feature_used_bytree()[feature_index]) continue;
327     if (parent_leaf_histogram_array_ != nullptr
328         && !parent_leaf_histogram_array_[feature_index].is_splittable()) {
329       smaller_leaf_histogram_array_[feature_index].set_is_splittable(false);
330       continue;
331     }
332     is_feature_used[feature_index] = 1;
333   }
...

Q: In the model file of LightGBM, it shows “num_leaves=63” in every iteration. Shouldn’t it change the depth and leaves of model for every iteration?

A: Can’t answer yet. Still need to look into the code to see why…