[返回] [最新] [最热门] [最高评价]

Trace memory error of CUDA program

The program which used CUDA for computing in GPU reported error about memory:

terminate called after throwing an instance of 'std::runtime_error'
what(): [CUDA] an illegal memory access was encoun
...

ROBIN DONG 2021-05-14 08:57 | 查看: 37

Migrate Spark job to BigQuery

I have just finished a work about migrating Spark job to BigQuery, or more precisely: migrate Python code to SQL. It’s a tedious work but improve the performance significantly: from 4 hours runt
...

ROBIN DONG 2021-05-07 08:45 | 查看: 31

Take care of the comma (in Python)

Think about the result of this snippet:

def concat(a, b):
return a + "_" + b

left = "hello",
right = "world"

print(concat(left, right))

Should be “hello_wor
...

ROBIN DONG 2021-04-29 12:06 | 查看: 37

Debug CUDA error for PyTorch

After I changed my dataset for my code, the training failed:

/tmp/pip-req-build-_tx3iysr/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:310: operator(): block: [0,0,0], thread: [59,0,0] Assertion `
...

ROBIN DONG 2021-04-23 09:17 | 查看: 30

Be careful when you use “isin()” method in Pandas

import pandas as pd

df_excl = pd.DataFrame({"id": ["12345"]})
df = pd.DataFrame({"id": ["12345", "67890"]})

result = df[~df.id.isin(df_excl[["i
...

ROBIN DONG 2021-04-09 12:17 | 查看: 27

An error about multiprocessing of Python

Our python program reported errors when running a new dataset:

[77 rows x 4 columns]]'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647",)'
multiprocessing.poo
...

ROBIN DONG 2021-04-08 08:56 | 查看: 34

Source code reading of LightGBM

Finally I get a few hours to look into the code of LightGBM.

I used to have some questions about LighGBM, and now fortunately I can answer some of them by myself. Even some answers may be wrong, that
...

ROBIN DONG 2021-03-31 11:54 | 查看: 18

Accelerate reading of NumPy array from files

In the training process, I need to read array data from .npy file and get a part of it:

import numpy as np

data = np.load("sample1.npy")
sound1 = data[start1: end1]
sound2 = data[start2: e
...

ROBIN DONG 2021-03-19 07:18 | 查看: 22

Change the schema of BigQuery tables

We can easily add new column for a table in BigQuery:

ALTER TABLE mydataset.mytable
ADD COLUMN new_col STRING

But when you want to delete or rename an existed column, there is no SQL to implem
...

ROBIN DONG 2021-03-11 11:27 | 查看: 16

Strange time output in a container of Kubernetes cluster

After running a workflow in Argo, I found out the output of the “date” command is totally wrong:

# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# T
...

ROBIN DONG 2021-03-05 08:49 | 查看: 34

Some tips about Python, Pandas, and Tensorflow

There are some useful tips for using Keras and Tensorflow to build models.

1. Using applications.inception_v3.InceptionV3(include_top = False, weights = ‘Imagenet’) to get pretrained para
...

Robin Dong 2019-02-06 10:47 | 查看: 4567

LinearSVC versus SVC in scikit-learn

In competition ‘Quora Insincere Questions Classification’, I want to use simple TF-IDF statistics as a baseline.

def grid_search(classifier, parameters, X, y, X_train, y_train, X_test, y_
...

Robin Dong 2019-01-26 11:32 | 查看: 1287

...更多...