- Get the memory size of a DataFrame of Pandas
Python
x
1
1
df.memory_usage(deep=True).sum()2. Upload a large DataFrame of Pandas to BigQuery table
If your DataFrame is too big, the uploading operation will report “UDF out of memory”
Console
1
1
1
google.api_core.exceptions.BadRequest: 400 Resources exceeded during query execution: UDF out of memory.; Failed to read Parquet file [...]. This might happen if the file contains a row that is too large, or if the total size of the pages loaded for the queried columns is too large.The solution is as simple as splitting the DataFrame and upload them one by one:
Python
1
6
1
client = bigquery.Client()2
for df_chunk in np.array_split(df, 10):3
job_config = bigquery.LoadJobConfig()4
job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND5
job = client.load_table_from_dataframe(df_chunk, table_id, job_config=job_config)6
job.result()3. Restore table in BigQuery
How to recover a deleted table in BigQuery? Just use bq command
Bash
1
1
1
bq cp dataset.table@1577833205000 dataset.new_tableIf your <timestamp> is not correct, the bq command will give you a notification about what <timestamp> is right for this table. Then you can use that correct <timestamp> again.